I am developing a website somewhat like leetcode, however, it will be for website testing using Selenium WebDriver.My service will provide different websites, along with various tasks that describe what to test for on the provided websites. The users will write selenium scripts, which my backend will compile and execute, and return the results to the user, specifying whether their tests passed or failed.The part I can’t seem to figure out is how I will check if the submitted user solution is correct or wrong.
I need suggestions on how to achieve this, as I am not very knowledgeable to the different things that can be possible here.
Here is my understanding of how leetcode checks solutions submitted by users. All of leetcode questions are about data structures and algorithms. In this case, each function has some input, and an output. Leetcode runs the function with test data as input, and compares the output to the expected output. If the output is same, the solution passes, otherwise, it fails.
In the case of testing, however, the methods have no input or output. The test methods are run by a test runner, and based on the assert statements, the test runner tells us whether the test passed or failed.
We cannot purely rely on the test runners verdict, as a user can simply put
assert.equal(true,true); and the test will pass, no matter what the test question is.
Or in other case, the user might perform some other actions, different from what the questions asks, and the test still passes.
Here is an example task and a solution(C#, NUnit) I intend to have on my website.
Points for task: 5
Task Name: Confirm addition of a product to cart.
Task description: Given an e-commerce website that sells clothing items, your task is to add the fleece jacket to the cart. After this, you must check that the Remove from cart button appears.
Website under test: www.saucedemo.com
The user solution might be the following test method
public void AddFleeceJacketToCart()
IWebElement AddFleeceJacketButton = driver.FindElement(By.CssSelector("button[data-test='add-to-cart-sauce-labs-fleece-jacket']"));
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(2));
IWebElement RemoveFleeceJacketButton = wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementIsVisible(By.CssSelector("button[data-test='remove-sauce-labs-fleece-jacket']")));
My Ideas for checking the sumbitted solutions:
Keywords : During task creation, a list of keywords are specified, which the solution will be checked against. These are keywords that must be part of the test. Keywords that should not be part of the test are also specified. For example, checking if the correct URL exists in the code, the necessary elements to work with, checking for appropriate actions, other things that can be exploited, like tricks with assert statements.
Logs : Using Selenium’s functionalities like command listeners, or directly inject event listeners into the browser, we can log all the events that take place during the test. We can check these logs automatically and compare them to the expected ones.
Question: what is a test supposed to do?
Answer: verify that the system under test (website in this case) has the desired functionality.
Therefore the answer is relatively obvious: you run the test both in situations where it is known to work and in situations where it is known to fail. Passing when it is supposed to fail is just as bad (if not worse) than failing when it is supposed to pass.
The key question here is “do you have control of the website that’s being tested?”
If you don’t I’m not sure it’s realistically possible, You would have to analyse and virtually duplicate the site or something.
Assuming you do control the site.
Change the required test output.
The problem as stands is you want the test to do multiple things, run some actions, verify some state, return a boolean pass/fail depending on the state. But it only outputs one thing. true/false
Add an additional requirement that it write to console is a machine readable format that you specify.
The userid returned after a successful login
The product id of the fleece jacket
The order id of the item in the basket
and whatever else can be verified
Specify that the test must take the url of the site as an input
Setup several versions of the site, each designed to fail at various points
One where the login will fail
One without the fleece
One with no add button
One with no remove
One which works as expected
Each version should return static values for the various things that you have asked the user to log. ie always the same userid, orderid etc
- Run the test against each
- Compare the output of each run against the output of a known good test.
- Score the test on how well it did