Summary:
exactness_play -s -t 1.exu -o 2.exu
The 2.exu has only the screenshot and image.
If it contain the binary code information, 2.exu becomes completed exu.
Test Plan:
exactness_play -s -t 1.exu -o 2.exu
and
change some font(elementary_config)
and
test in ea_gen tool
Reviewers: JackDanielZ
Reviewed By: JackDanielZ
Subscribers: myoungwoon, akanad, bowonryu, YOhoho
Differential Revision: https://phab.enlightenment.org/D7472
Summary:
The way it was done until now doesn't involve all the EFL layers, such
as Edje.
Now, we support the click event that is converted to a series of three actions { move,
up, down } appended to the played actions list.
Reviewers: jsuya
Reviewed By: jsuya
Subscribers: bowonryu, akanad, myoungwoon, YOhoho
Differential Revision: https://phab.enlightenment.org/D7386
Summary:
Because the environment variables were set on the player and not on the
application, they were not affecting the application, leading to issues
when the EFL configuration scaling is different of 1.0.
By appending them right before the application name, they are taken in
consideration, fixing the issue.
Reviewers: jsuya, YOhoho
Reviewed By: jsuya
Differential Revision: https://phab.enlightenment.org/D7388
Summary: No exu is created if no diffs have been found. However, if diffs are present, we append all the comparison images, including those which don't have differences.
Differential Revision: https://phab.enlightenment.org/D7355
Summary: This is helpful to get simple result of .exu diff without file access
Test Plan: exactness_inspect --compare test.exu test.compare.exu --output=out.exu
Reviewers: jsuya, JackDanielZ
Reviewed By: JackDanielZ
Subscribers: myoungwoon, akanad, bowonryu
Differential Revision: https://phab.enlightenment.org/D7239
Now that exu are supported, it is better that the player deals with the
destination management, including removing the old shots if the
destination is a directory.
In most scenarios, a lot of time is useless and taken by the human
inaction or slowness.
This optimizer tries to increase the scenario speed by removing this
wasted time. Stabilization is required before shots.
Note that animated tests such as label_slide cannot be optimized well.
The reason is that a stabilization is done at the beginning of the test,
resulting in an endless loop (slide always moving).
When changes are made in the source actions list (stabilize-shots...),
the modifications will be stored in the exu destination. This is not what we
want.
This patch fixes it by storing the original source scenario into the
destination exu.
It can happen that some tests take more time on weaker machines, leading
to wrong screenshots.
By using this option, the player will wait before each shot for the
application to be stabilized (no mode frames changes) and then take the shot.
It can happen if the file doesn't contain any actions or if no shot is
present. In this last case, an optimization in the player removes the
actions after the last shot, meaning it will remove all the actions if
no shot is present.
An example is window_states_2 test where the shots cannot be taken by
the recorder, as F2 is not caught.
To compare two exu files during the test suite, we need to unpack
the images in directories and make the comparison from there.
The problem is that the "orig" exu file was unpacked in the current
directory as well, leading to compare an image with itself.
The fix consists in unpacking the right exu file.
This can be useful if the application is not in an official repository
like elementary_test.
This feature allows compiling the source code and using it to run the
scenario on.
The command (e.g gcc...) must be stored in the unit as well. $SRC and
$DEST can be used to specify the source and destination files in the
command.
When the external injection is used (scenario injected by an external
tool via Eina Debug) and a screenshot is required, the data is sent
to the injector.
A bug appears in "box vert" test. If --disable-screenshots is set, and
no other optional such as -v, the application will receive only 4
clicked events instead of 8. The bug doesn't appear if -v is set or if
the application is shown on the screen.
It seems that the buffer engine doesn't want to consume the events
because no damage/shot request is "reported".
The solution for that bug is to enable the screenshots process, just
without saving the outputs.
@fix
Buffer evas cannot be used when the application is shown on the screen.
It means the _sync_shot_get function cannot work always. That is a
problem when the stabilization feature is used.
That's why we need to use snapshot no matter the case. The problem is
that snapshot seems to sit on the canvas and therefore doesn't forward
the events, such as mouse down... So we need to create a snapshot only
when needed.
Additionally, stabilization feature needed to be finished. The threshold
to guarantee the stabilization was not implemented.
One of the biggest issue in Exactness is related to the system
configuration differences. Among them, the fonts can for example
impact on the height of the widgets.
The solution to not be dependent on the fonts consist in using embedded
fonts and to force their usage when playing the applications.
The -f option has been added to the player and the recorder so the user
can provide the path to a fonts directory. This option must be set in
order to force the fonts replacement. Since tests shots can use different
fonts, the exu file stores the version of fonts that have been used.
This is why it is needed to have in the provided directory different
directories, each pointing to a different version of the fonts. For
example, some old tests can use fonts of 2017 (e.g directory 20170101)
while new tests will use new fonts (20180601). Check the
exactness-elm-data repository (fonts branch) for a better understanding.
During recording, the -f option will apply the indicated fonts on the
launched application and will record the mouse events accordingly. The
fonts datestamp is stored in the exu output.
During playing, the fonts will be loaded by reading the exu fonts path,
and then the application is launched. If no information is provided in
the exu but -f is used, the tool will load the most recent fonts (by
comparing the datestamp directories).
It is still on going.
One of the issues is that for some mystic reason, the
HAVE_EXACTNESS_DATA seems not checked as the target check-exactness is
always present, no matter if the submodule exists.
The problem that we try to solve is the time taken to compare the shots after
all the scenarios have been run.
Now, comparing the shots sequentially is done right after the test
finished to run. With -j 1, it won't change anything. With more CPUs, it
will compare while other tests are running, i.e when the CPU is not too
much busy.
The goal is to support applications where editable entries are used.
The problem is the text cursor that, even if we disable its animation
through the theme overlay, triggers the render post event, which breaks
all the previous method used to detect stability.
Now, every 100ms, we compare the current canvas image with the previous
saved shot.
junsu choi
Daniel Zaoui
Yeongjong Lee