Commit Graph

231 Commits

Author SHA1 Message Date
Daniel Zaoui ede4ff100e Inspect: create a full comparison output if exu is required and diffs are present
Summary: No exu is created if no diffs have been found. However, if diffs are present, we append all the comparison images, including those which don't have differences.

Differential Revision:
2018-11-24 21:45:47 +02:00
Daniel Zaoui 239e38f173 Player: reset evas pointer inside list on deletion 2018-11-18 15:47:11 +02:00
Yeongjong Lee 12af68f993 Inspect: support to print the exu diff result to standard output
Summary: This is helpful to get simple result of .exu diff without file access

Test Plan: exactness_inspect --compare test.exu --output=out.exu

Reviewers: jsuya, JackDanielZ

Reviewed By: JackDanielZ

Subscribers: myoungwoon, akanad, bowonryu

Differential Revision:
2018-11-11 11:33:49 +02:00
Daniel Zaoui 433d843898 Suite: add in live status per test
In this way, a program launching the suite can easily parse the messages
and update statuses in a dedicated GUI.
2018-11-07 15:01:57 +02:00
Daniel Zaoui f7dfae4423 Inspector: implement command line exu comparison
Only png comparison was supported.
2018-11-04 10:59:11 +02:00
Daniel Zaoui e48bda302a Player: permit speed increasing 2018-11-04 10:59:11 +02:00
Daniel Zaoui 82cfa8378a Inspector: fix clean mode
Even if duplicate actions are removed, the delay should remain the same
between two actions, no matter how many actions have been removed
2018-11-04 10:59:11 +02:00
Daniel Zaoui da73209747 Inspector: display image in real size when selected 2018-11-04 10:59:11 +02:00
Daniel Zaoui b2cf106517 Move scenario cleaning from player to inspector 2018-11-04 10:58:54 +02:00
Daniel Zaoui 89a207bd34 Inspect: permit packing given scenario and images into a exu file 2018-11-04 10:58:51 +02:00
Daniel Zaoui 6f74e380a4 Remove old shots from inside the player
Now that exu are supported, it is better that the player deals with the
destination management, including removing the old shots if the
destination is a directory.
2018-11-04 10:56:35 +02:00
Daniel Zaoui 9506ce55e8 Suite: fix scenarios lookup when many paths are given 2018-11-04 10:56:35 +02:00
Daniel Zaoui 4890fd01f7 Revert "Inspector: add optimize option"
This reverts commit b95961ba0c.

The optimization needs more work.
2018-11-04 10:56:35 +02:00
Daniel Zaoui b5748fa921 Inspector: add action removal feature
This feature permits removing actions from exu files
2018-10-29 19:58:55 +02:00
Daniel Zaoui b95961ba0c Inspector: add optimize option
In most scenarios, a lot of time is useless and taken by the human
inaction or slowness.

This optimizer tries to increase the scenario speed by removing this
wasted time. Stabilization is required before shots.

Note that animated tests such as label_slide cannot be optimized well.
The reason is that a stabilization is done at the beginning of the test,
resulting in an endless loop (slide always moving).
2018-10-29 10:04:28 +02:00
Daniel Zaoui 3d02d1c50e Player: use the source scenario for the exu destination
When changes are made in the source actions list (stabilize-shots...),
the modifications will be stored in the exu destination. This is not what we
This patch fixes it by storing the original source scenario into the
destination exu.
2018-10-29 00:07:10 +02:00
Daniel Zaoui 4fc682df84 Player/stabilize: fix duplicate actions
If a stabilize action is already present before a shot of an exu, the
stabilize-shots option should not append another one.
2018-10-29 00:05:21 +02:00
Daniel Zaoui a576265f0a Player: add an option to wait for the frames to be stable before shots
It can happen that some tests take more time on weaker machines, leading
to wrong screenshots.
By using this option, the player will wait before each shot for the
application to be stabilized (no mode frames changes) and then take the shot.
2018-10-21 00:14:55 +03:00
Daniel Zaoui b6bd7ecdf0 Player: fix crash when no action is present in the file
It can happen if the file doesn't contain any actions or if no shot is
present. In this last case, an optimization in the player removes the
actions after the last shot, meaning it will remove all the actions if
no shot is present.

An example is window_states_2 test where the shots cannot be taken by
the recorder, as F2 is not caught.
2018-10-15 11:43:20 +03:00
Daniel Zaoui 3f31c8f72e Suite: fix exu files comparison
To compare two exu files during the test suite, we need to unpack
the images in directories and make the comparison from there.
The problem is that the "orig" exu file was unpacked in the current
directory as well, leading to compare an image with itself.

The fix consists in unpacking the right exu file.
2018-10-03 12:25:50 +03:00
Daniel Zaoui 9831560caf Use source codes stored in the unit as application
This can be useful if the application is not in an official repository
like elementary_test.
This feature allows compiling the source code and using it to run the
scenario on.
The command (e.g gcc...) must be stored in the unit as well. $SRC and
$DEST can be used to specify the source and destination files in the
2018-08-29 12:08:22 +03:00
Daniel Zaoui 5ffc8ec640 Player: send back screenshot data when remotely requested
When the external injection is used (scenario injected by an external
tool via Eina Debug) and a screenshot is required, the data is sent
to the injector.
2018-08-22 21:59:42 +03:00
Daniel Zaoui 1551b5358f Player: enable shots (silently) even if disabled in the command line
A bug appears in "box vert" test. If --disable-screenshots is set, and
no other optional such as -v, the application will receive only 4
clicked events instead of 8. The bug doesn't appear if -v is set or if
the application is shown on the screen.

It seems that the buffer engine doesn't want to consume the events
because no damage/shot request is "reported".

The solution for that bug is to enable the screenshots process, just
without saving the outputs.

2018-07-11 12:18:52 +03:00
Daniel Zaoui d8315ffdc7 Remove printfs 2018-06-10 12:33:35 +03:00
Daniel Zaoui 49e455f481 Player: try to stabilize screenshotting
Buffer evas cannot be used when the application is shown on the screen.
It means the _sync_shot_get function cannot work always. That is a
problem when the stabilization feature is used.
That's why we need to use snapshot no matter the case. The problem is
that snapshot seems to sit on the canvas and therefore doesn't forward
the events, such as mouse down... So we need to create a snapshot only
when needed.
Additionally, stabilization feature needed to be finished. The threshold
to guarantee the stabilization was not implemented.
2018-06-10 11:58:40 +03:00
Daniel Zaoui e59cca6fce Suite: support player --disable-screenshots options 2018-06-10 11:57:28 +03:00
Daniel Zaoui 08a6abd881 Fix how the jobs are consumed
In init mode, we don't need to compare. So we have to check the mode and
then decide if a comparison or the next test execution is needed.
2018-06-07 12:46:32 +03:00
Daniel Zaoui 4d0579aa15 Fix command preparation in simulation mode
The sentinel was forgotten, leading to issues when launching the
2018-06-07 12:45:39 +03:00
Daniel Zaoui 0e1f0cf1c5 Exactness: add fonts support
One of the biggest issue in Exactness is related to the system
configuration differences. Among them, the fonts can for example
impact on the height of the widgets.
The solution to not be dependent on the fonts consist in using embedded
fonts and to force their usage when playing the applications.

The -f option has been added to the player and the recorder so the user
can provide the path to a fonts directory. This option must be set in
order to force the fonts replacement. Since tests shots can use different
fonts, the exu file stores the version of fonts that have been used.
This is why it is needed to have in the provided directory different
directories, each pointing to a different version of the fonts. For
example, some old tests can use fonts of 2017 (e.g directory 20170101)
while new tests will use new fonts (20180601). Check the
exactness-elm-data repository (fonts branch) for a better understanding.

During recording, the -f option will apply the indicated fonts on the
launched application and will record the mouse events accordingly. The
fonts datestamp is stored in the exu output.
During playing, the fonts will be loaded by reading the exu fonts path,
and then the application is launched. If no information is provided in
the exu but -f is used, the tool will load the most recent fonts (by
comparing the datestamp directories).
2018-06-05 00:06:18 +03:00
Daniel Zaoui 621dac67ce Fix uninitialized variables 2018-06-04 23:37:41 +03:00
Daniel Zaoui e215368501 Replace arrays with alloca to prevent dumb warnings
snprintf now throws warning during compilation on potential buffer
overflow when using arrays.
2018-06-04 23:37:35 +03:00
Daniel Zaoui 4203336691 Remove references to POSIX 2018-06-04 13:38:11 +03:00
Daniel Zaoui 4b5ced9b07 Fix potention buffer overflow 2018-06-04 13:35:18 +03:00
Daniel Zaoui ab22f74b0a Fix warnings
n_evas is unsigned so always above or equal 0
2018-06-04 12:23:31 +03:00
Daniel Zaoui b95c50de65 Player/injection: don't feed events from an internal source 2018-06-04 12:22:31 +03:00
Daniel Zaoui e13c15f39e Fix warnings related to string manipulation 2018-06-04 12:20:50 +03:00
Daniel Zaoui 55cba900b9 Revert "Add exactness-elm-data as submodule"
This reverts commit 3ddd87589d.

This patch should not have been pushed.
2018-06-04 12:11:53 +03:00
Daniel Zaoui 65c07efc0e Revert "Add check-exactness target with -j support"
This reverts commit f1bfcaeaf0.

This patch should not have been pushed.
2018-06-04 12:11:31 +03:00
Daniel Zaoui ee532a6aac Inspect: fix extension extraction when different extensions are given
When a exu and a rec file were given, the ext was remaining exu and a
crash was happening.
2018-05-23 09:35:00 +03:00
Daniel Zaoui f1bfcaeaf0 Add check-exactness target with -j support
It is still on going.

One of the issues is that for some mystic reason, the
HAVE_EXACTNESS_DATA seems not checked as the target check-exactness is
always present, no matter if the submodule exists.
2018-05-22 22:35:39 +03:00
Daniel Zaoui 3ddd87589d Add exactness-elm-data as submodule 2018-05-22 22:35:08 +03:00
Daniel Zaoui 92457031ee Add first support for Windows and fix many other config stuff
Thanks to vtorri for this patch
2018-05-20 21:29:45 +03:00
Daniel Zaoui 7337672562 Compare the shots directly after the test to speed up
The problem that we try to solve is the time taken to compare the shots after
all the scenarios have been run.
Now, comparing the shots sequentially is done right after the test
finished to run. With -j 1, it won't change anything. With more CPUs, it
will compare while other tests are running, i.e when the CPU is not too
much busy.
2018-05-20 21:29:45 +03:00
Daniel Zaoui a1341ac74c Fix Valgrind issue
It was complaining of an invalid read size because it was accessing
bytes after the pixels buffer.
2018-05-18 17:21:13 +03:00
Daniel Zaoui 667171d1d9 Player: improve the stabilization mechanism
The goal is to support applications where editable entries are used.
The problem is the text cursor that, even if we disable its animation
through the theme overlay, triggers the render post event, which breaks
all the previous method used to detect stability.
Now, every 100ms, we compare the current canvas image with the previous
saved shot.
2018-05-16 18:00:39 +03:00
Daniel Zaoui 8eafc850bc Player: don't play actions after the last shot
There is no good reason to play these actions, as no comparison is made
2018-05-16 18:00:39 +03:00
Daniel Zaoui f12aa93376 Inspect: permit stabilization action appending via command line 2018-05-16 18:00:39 +03:00
Daniel Zaoui 3457071db1 Recorder: support stabilization via hotkey 2018-05-16 18:00:39 +03:00
Daniel Zaoui f56ee0dfac Suite: replace the compare method
Instead of using ImageMagick, we now use an internal function.
2018-05-16 18:00:39 +03:00
Daniel Zaoui 53a52a3419 Suite: support exu files comparison 2018-05-16 15:16:24 +03:00