System test with robot framework (#8481)
* prototype sysTest
Requires:
robotFramework - the system test framework
pyautogui - for controlling keyboard/mouse
To install:
pip install robotframework
pip install pyautogui
* WIP include flaky way of exiting
Ensure this is run from a cmd prompt. Other shells may cause issues,
for instance, NVDA does not get correct focus when run from cygwin.
* WIP System testing
Remote server implemenation and external abstracted library.
* First try at including system tests in appveyor.
* Disable pyAutoGUI's failsafe check as appeyor moves the mouse???
* initial.robot: break robot to temporarily bipass freeze.
* appveyor.yaml: run robot in powershell so exit code is ignored for now.
* appveyor.yaml: try again to make system tests return error code 0
* appveyor.yaml: try again to make system tests return error code 0
* appveyor.yaml: fix syntax of uplodFile.
* system tests: tell appVeyor they are junit formatted?
* appVeyor: still upload test results on failed builds.
* Upload unitTest reports to appViewer as well.
* Investigate freeze in robot test.
* add some logging to nvdaRobotLib
* More logging
* more logging again
* sleep before connecting to server.
* more and more logging
* Fix logging...
* reimplement quit nvda test inside initial.robot.
* A try at moving everying to robot language to fix freezes.
* Launch NVDA from robot with py for appveyor.
* copy the globalPlugin.
* Temporarily start rdp for appveyor build
* bump
* bump 2
* bump 3
* bump 4
* Slightly re-write now with setup and teardown, and insert q for quit check and check foreground name is Exit NVDA etc.
* nvdaRobotLib: use a specific NVDA profile dir.
* NVDA Spy for robot must use a daemon thread for now to stop further NVDAs crashing.
* Robot tests set NVDA not to show the welcome dialog
* allow asserting on last speech
* Remove extra tab characters
* Copy in the testSpy globalPlugin for each test.
* Remove sleep
The sleep was required because importing the remote library suceeds
regardless of whether it is possible to execute a keyword. We now test
the connection before we load the library.
* Some cleanup of connection/disconnection code
* Extract code that can be reused and add better documentation
* cleanup systemTest dir
Allow use of different settings files and some cleanup of system test directories
* Assert text on welcome dialog passes
* Copy nvda log for each test
* Try to get systemtest artefacts from appveyor
* Try manually pushing artifacts
* Try recursing over files and pushing manually
* Fix issue when nvdaProfile folders do not exist
* system tests: don't show usage stats dialog in NVDA
* appveyor: install NVDA on the build system, and run the system tests against the installed copy.
* appveyor: install nvda with start-process and wait up to 3 minutes for it to complete, failing on timeout.
* fix appveyor errors
* Temprarily include robotremoteserver in NVDA for system tests. Must figure out a better solution.
* Breakup up appveyor test code into separate powershell code chunks to better show build progress.
* system tests: use silence synth driver.
* system test for exit dialog: just sleep instead of varifying dialog.
* system tests: don't wait for exit dialog, rather just confirm that the NVDA process definitlly exited.
* appveyor: move NVDA installation to before_test section.
* ensure quit system test: exit from the NVDA menu.
* system tests: try alt tabbing to the exit dialog???
* More debugging for system tests.
* IAccessibleHandler: more debugging.
* IAccessibleHandler: more debugging.
* eventHandler: more debugging.
* Up max foreground defer count to 10.
* Up foreground defers to 100 and log them.
* IAccessibleHandler: drop foreground defers back down and allow welcome dialog in quit from keyboard system test.
* system tests for quit with keyboard: watch for and close welcome dialog first.
* quit with keyboard system test: wait longer for NVDA to exit.
* Restore IAccessibleHandler and eventHandler.
* Not able to get an index returned from get_index_of_speech
* tests passing
* move spy lib logic into spy
* Install systemTestSpy as a package on setup
* Cleanup files, add copyright headers
* Use _ to prevent imports from leaking as keywords
* Move non NVDA specific helpers into a new library
* Create self contained package for system test spy
The system test spy now copies in library dependencies that are not
provided by a NVDA install. This is so that the system tests can be run
on appVeyor against an installed version of NVDA
* Zip system test output before pushing to artifacts
* try adding robotlibraries directory to the path
* Fix case of file name
This should resolve the appveyor system test failures
* further tidy up
* Introduce suite name
* fix appveyor system tests
testOutput directories need to be created before writting files to them!
* Stop double nvda install log artifact
* set robot library path
* reduce suite verbosity on appveyor
* Revert to running from repo root
Running from tests/system/ did not reduce the number of suites mentioned in Robots reports
* Clean up some of the appveyor code
* clean up some names and docs for tests
* Add scons command to run system tests
* ensure dir is created for std out/err logs
* Use dummy speech synth to get speech
This captures all NVDA manipulation of speech before it is given to
synthesizers.
* wait longer before sending input
* Stop update dialog from showing on startup
* simplify the appveyor pip installs
* Update What's new
