You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Tom Hacohen 33df8ecb2c Get rid of the compare option, useless. 10 years ago
data Exactness: Compilation is fun, lets make it do that. 11 years ago
src Get rid of the compare option, useless. 10 years ago
.gitignore Exactness: added .gitignore. 11 years ago
AUTHORS Exactness: initial commit. 11 years ago
COPYING Exactness: initial commit. 11 years ago
ChangeLog Exactness: initial commit. 11 years ago
INSTALL Exactness: initial commit. 11 years ago
Makefile.am Exactness: install recordings. 11 years ago
NEWS Exactness: initial commit. 11 years ago
README exactness: Make it possible to run only selected tests from list file. 11 years ago
autogen.sh Exactness: initial commit. 11 years ago
configure.ac trunk: remove use of AM_PROG_CC_STDC as AC_PROG_CC does it. 10 years ago
tasn_is_a_lazy_bastard revert unrelated change from 65961 11 years ago

README

exactness is a software package aimed to automate Elementary testing
after updating elm code.
The testing process is composed of running widget test,
cycling through API functions, and screenshot are taken on each stage of test.
(on user request with F2 key).

Next, these screenshots are compared with screenshots of widget test
done before the code update.
When exactness detects difference in screenshot comparison,
it means widget layout was changed.
Unexpected layout changes may reveal that something is broken
in the new implementation.

Testing with exactness composed of the following stages:
First, you need to compose a tests (text) file as follows:
Each line begins with test name
second field is test-command and [optional] params.
Any line starting with '#' is a comment (ignored):

# This is a comment line
TestName1 TestCmd [param1] [param2]

Any name/parameter with space, precede space with double backslash (\\)
Example:
pager_slide elementary_test --test-win-only pager\\ slide
In this example test name is "pager"
The application is "elementary_test"
with two parameters: --test-win-only pager\\ slide
("pager slide" compose singe parameter, note the \\)

By default, exactness runs through test file and commits all tests specified.
To run selected tests add test name as param to exactness.
More on this later.

Later, you run exactness with the tests file as parameter.
1. Test Recording.
Done once and exactness user most likely does not need to do this.
A set of pre-recorded tests already resides under 'data/recording' folder.

To record a test just issue the command:
exactness -r [-b BaseDir] TestsFile

Example:
exactness -r TestFileName

The record file will be produced in the current-working-directory
for each line in the test file.
Name of '.rec' file determined according to test name:
From example on top we have: pager_slide.rec

To save your recordings in any other directory,
pass the directory-name as base-dir:
exactness -r -b /home/username/my_recordings TestFile

As test runs you may press F2 at any point to set a screenshot.
(screenshot produced in 'play' stage, set timing of screenshot here)
However, don't do this while animation is ongoing
or you get inconsistent PNG output each time test played.

NOTE: You may change screenshot key from F2 to any other key
with TSUITE_SHOT_KEY env var.


After recording a new test you may use 'simulate' option to play test
with no effect to output files. Simulate option lets you to examine
a test-play on-screen without actually producing output PNG file.

To do this just run:
exactness -s [-b BaseDir] TestFile

Example:
exactness -s TestFile
This will play all tests specified in TestFile taking '.rec' files in cwd.
use the -b option to tell exactness to locate '.rec' files elsewhere.


2. Test Initiation.
The final stage of testing process is based on PNG image comparison
of original screenshots with current screenshots produced after code change.

To start you need to produce original screenshot set before code change.
To do that just issue the command:
exactness -i [-b BaseDir] TestFile

This command will play the tests from TestFile and place PNG files
under 'orig' folders created in current-working-directory.
You need to do this just once.

exactness -i TestFile
(locate '.rec' files in cwd)

This will produce PNG files such orig/pager_slide_1.png

By default exactness looks for '.rec' files in current-working-directory
Use -b DirName to ask exactness to locate the '.rec' files
in a specific location.

Example:
exactness -i -b ${prefix}/share/exactness/recordings TestFile
(locate '.rec' in "${prefix}/share/exactness/recordings")


3. Test playing, producing current state PNG files and comparison files (4).
After doing Test Initiation (2) and updating elementary code,
you may run tests for current state of elm:

exactness -p [-b BaseDir] [-d DestDir] TestFile
Omitting DestDir param will cause exactness place all PNG files
under (default) folder named 'current'.
Note that comparing PNG files (4) is also done upon completion of play.
This will produce comp_*.png files in your DestDir.

You may put record-files at specific folder with -b (base dir) argument.
Example:
exactness -p -b ${prefix}/share/exactness/recordings TestFile

Play '.rec' files at "${prefix}/share/exactness/recordings"
Write resulting PNG files at cwd/current (default) dir
along with comp_*.png files.


4. Comparing PNG files.
This is the final stage of testing in which we compare output
before code update and after the update.
Following 'play' action, matching PNG files from 'orig' and 'current'
folder are compared.

When all comparisons done you get final report of (N) files did not match.
In case difference was found, you may look at comp_*.png files in DestDir.


5. fail_if(expr) macro:
You may use fail_if(expr); macro to verify variable value on-the-fly.
For example, if your code contains:

char *p;
...
fail_if(p == NULL);

In case (p == NULL), a "fail_TestName.txt" file is crated
current working directory with a comment specifying the expr/line failed.

fail_*.txt files are produced only for when expr fails when running.
When tests completed, the user gets notification message if any tests failed.
In such case, please review the fail_*.txt files.

6. Running selected tests.
To run selected tests add test name as param to exactness.
Use this option to run selected tests without modifying your test file.
Usage:
/opt/e17/bin/exactness -s TestsFile TestName1 [TestName2] [...]
TestName param has to match test name given in tests file (1st field)


7. To view exactness help issue the command:
exactness --help
or
exactness -h