summaryrefslogtreecommitdiff
path: root/README
blob: 5d0cd755bdf40e5631270b594968b3333915a6be (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
Exactness is a pixel perfect regression testing tool for Elementary based applications.

Exactness works by playing back a recording of an interaction with the
application, saving images from that playback, and comparing the images to the
set of "known to be correct" images. Returning a non-zero return value when it
detects images mismatches and creates "error.html", a file that includes the
list of tests failed, including the base images, the new images, and a visual
"diff" of the images. Exactness also fails in case the tests themselves exit
with a non-zero return value, e.g segfault.

= Modes of operation =

Exactness has four modes of operation, three of which are essential for normal
operation, and the fourth is helpful in some cases, and in general, just pretty.

== List of Modes ==

# record (-r, --record): record a test macro.
# init (-i, --init): run the test macro and take images of the state.
# play (-p, --play): run the test macro, take images of the state, and compare against the base images (took with init).
# simulation (-s, --simulation): run the test macro, showing the process on screen.

= The List file =

The list file contains a list of tests to actually run.
The file format is:
<test name> <test command>
<test name> has to be one word (no spaces), there are no restrictions on the command. The command is executed using "system", so you can set environment variables and generally do whatever you'd like.
Empty lines and lines starting with # are ignored.

Example line:
  box ./elm_edje_player tests/box.edj

= Using exactness =

For the rest of this section we'll assume the name of the list file is
"tests.txt".

== Recording ==

The first thing to do when running exactness on a new test is generating the
recording file, this is done but running with "-r". New recordings overwrite
old existing recordings so it's always a good idea to backup recordings you are
happy with.
In order to record for a test list run:
  exactness -r tests.txt
Just interact with the application in any way you'd like (it's better to have
short interactions as tests run in real time, so running the tests will be as
slow as your recording).
Press F2 to mark an image taking opportunity (as many times you'd like), this
will cause "init" and "play" mode to take images for comparison. Please refer
to the "Limitations" section below to read more about image taking
opportunities.
Running this will create the "base dir" (see command line options for more info
about how to change it) and populate it with one recording file per test.

== Init ==

After having a recording file for a test you can go on with capturing the base
images. Those images are assumed to be "correct". You should update them when
this assumption is no longer valid (e.g fixed a bug, updated a test).
Run:
  exactness -i tests.txt
This will look for recording files in the "base dir", run them, take images of
the application, and put them in "dest-dir/orig".

== Play ==

This is the most common mode of execution. This should be run on a regular
basis, for example in a CI system.
Run:
  exactness -p tests.txt
This will look for recording files in the "base dir", run them, take images of the application, put them in "dest-dir/current", and compare them against the
images in "dest-dir/orig". This will fail (and return a non-zero return value)
if any of the tests return a non-zero return value, or if any of the images
in the "current" dir don't match the ones in "orig" dir.
Detecting a failure is the most important role of exactness, so the return
value is very important. Additionally, an error.html file will be created in
the "current" dir, listing the issues in a human-readable form.

== Simulation ==

This mode is a show off and a debug mode. Very useful with the "-v" (verbose)
option that shows the mouse cursor on the canvas and outputs information to
console.
Run:
  exactness -s tests.txt
or for verbose mode:
  exactness -v -s tests.txt
This does nothing other than run the tests to screen. No images are taken,
and no report is generated.

== Command line options ==

* -b, --base-dir: The location of the recording files.
* -d, --dest-dir: The location of the base dir for the images (orig/current will be appended).
* -w, --wrap: Use a custom command to launch the tests with, e.g valgrind.
* -j, --jobs: Number of jobs to run in parallel. Very useful when running in init and play mode.
* -r, --record: record mode.
* -p, --play: play mode.
* -i, --init: init mode.
* -s, --simulation: simulation mode.
* -v, --verbose: Show cursor location and output extra information to console.
* -h, --help: Help, and information about more boring options.

= Limitations =

* Animations are very time-specific, you can't really take snapshots of
applications that have animations.
* Exactness is very susceptible to theme/scale/etc. changes, so make sure to
replicate the full environment.
* Only works with Elementary applications.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

REST IS POSSIBLY OUTDATED

exactness is a software package aimed to automate Elementary testing
after updating elm code.
The testing process is composed of running widget test,
cycling through API functions, and screenshot are taken on each stage of test.
(on user request with F2 key).

Next, these screenshots are compared with screenshots of widget test
done before the code update.
When exactness detects difference in screenshot comparison,
it means widget layout was changed.
Unexpected layout changes may reveal that something is broken
in the new implementation.

Testing with exactness composed of the following stages:
First, you need to compose a tests (text) file as follows:
Each line begins with test name
second field is test-command and [optional] params.
Any line starting with '#' is a comment (ignored):

# This is a comment line
TestName1 TestCmd [param1] [param2]

Any name/parameter with space, precede space with double backslash (\\)
Example:
pager_slide elementary_test --test-win-only pager\\ slide
In this example test name is "pager"
The application is "elementary_test"
with two parameters: --test-win-only pager\\ slide
("pager slide" compose singe parameter, note the \\)

By default, exactness runs through test file and commits all tests specified.
To run selected tests add test name as param to exactness.
More on this later.

Later, you run exactness with the tests file as parameter.
1. Test Recording.
   Done once and exactness user most likely does not need to do this.
   A set of pre-recorded tests already resides under 'data/recording' folder.

   To record a test just issue the command:
   exactness -r [-b BaseDir] TestsFile

   Example:
   exactness -r TestFileName

   The record file will be produced in the current-working-directory
   for each line in the test file.
   Name of '.rec' file determined according to test name:
   From example on top we have: pager_slide.rec

   To save your recordings in any other directory,
   pass the directory-name as base-dir:
   exactness -r -b /home/username/my_recordings TestFile

   As test runs you may press F2 at any point to set a screenshot.
   (screenshot produced in 'play' stage, set timing of screenshot here)
   However, don't do this while animation is ongoing
   or you get inconsistent PNG output each time test played.

   NOTE: You may change screenshot key from F2 to any other key
         with TSUITE_SHOT_KEY env var.


   After recording a new test you may use 'simulate' option to play test
   with no effect to output files. Simulate option lets you to examine
   a test-play on-screen without actually producing output PNG file.

   To do this just run:
   exactness -s [-b BaseDir] TestFile

   Example:
   exactness -s TestFile
   This will play all tests specified in TestFile taking '.rec' files in cwd.
   use the -b option to tell exactness to locate '.rec' files elsewhere.


2. Test Initiation.
   The final stage of testing process is based on PNG image comparison
   of original screenshots with current screenshots produced after code change.

   To start you need to produce original screenshot set before code change.
   To do that just issue the command:
   exactness -i [-b BaseDir] TestFile

   This command will play the tests from TestFile and place PNG files
   under 'orig' folders created in current-working-directory.
   You need to do this just once.

   exactness -i TestFile
   (locate '.rec' files in cwd)

   This will produce PNG files such orig/pager_slide_1.png

   By default exactness looks for '.rec' files in current-working-directory
   Use -b DirName to ask exactness to locate the '.rec' files
   in a specific location.

   Example:
   exactness -i -b ${prefix}/share/exactness/recordings TestFile
   (locate '.rec' in "${prefix}/share/exactness/recordings")
   

3. Test playing, producing current state PNG files and comparison files (4).
   After doing Test Initiation (2) and updating elementary code,
   you may run tests for current state of elm:

   exactness -p [-b BaseDir] [-d DestDir] TestFile
   Omitting DestDir param will cause exactness place all PNG files
   under (default) folder named 'current'.
   Note that comparing PNG files (4) is also done upon completion of play.
   This will produce comp_*.png files in your DestDir.

   You may put record-files at specific folder with -b (base dir) argument.
   Example:
   exactness -p -b ${prefix}/share/exactness/recordings TestFile

   Play '.rec' files at "${prefix}/share/exactness/recordings"
   Write resulting PNG files at cwd/current (default) dir
   along with comp_*.png files.
   
   
4. Comparing PNG files.
   This is the final stage of testing in which we compare output
   before code update and after the update.
   Following 'play' action, matching PNG files from 'orig' and 'current'
   folder are compared.

   When all comparisons done you get final report of (N) files did not match.
   In case difference was found, you may look at comp_*.png files in DestDir.

   
5. fail_if(expr) macro:
   You may use fail_if(expr); macro to verify variable value on-the-fly.
   For example, if your code contains:

   char *p;
   ...
   fail_if(p == NULL);

   In case (p == NULL), a "fail_TestName.txt" file is crated
   current working directory with a comment specifying the expr/line failed.

   fail_*.txt files are produced only for when expr fails when running.
   When tests completed, the user gets notification message if any tests failed.
   In such case, please review the fail_*.txt files.

6. Running selected tests.
To run selected tests add test name as param to exactness.
Use this option to run selected tests without modifying your test file.
Usage:
/opt/e17/bin/exactness -s TestsFile TestName1 [TestName2] [...]
TestName param has to match test name given in tests file (1st field)


7. To view exactness help issue the command:
   exactness --help
   or
   exactness -h