</p> <h1>RSelenium: Testing Shiny Apps</h1> <h2>Introduction</h2> <p>The goal of this vignette is to give a basic overview of how one might approach &ldquo;testing&rdquo; a shiny app. <a href="http://www.rstudio.com/shiny/">Shiny</a> is a new package from <a href="http://www.rstudio.com/">RStudio</a> that makes it dramatically easier to build interactive web applications with R. Shiny Uses a reactive programming model and has built-in widgets derived from the <a href="http://getbootstrap.com/javascript/">Bootstrap</a> front-end framework. In this vignette we will looking at writing unit tests for a simple shiny wep app. The testing package we will use is <a href="https://github.com/hadley/testthat">testthat</a> which has a short introduction <a href="http://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf">here</a>. I am using <code>testthat</code> version 0.8. The version on cran is version 0.7.1 and may give trouble for tests where I manipulate the test environment. You can install 0.8 from github <code>devtools::install_github(&quot;testthat&quot;, &quot;hadley&quot;)</code></p> <p>This vignette is divided into five main sections:</p> <ul> <li><a href="#id1">Some thoughts on testing.</a></li> <li><a href="#id2">The shiny test app.</a></li> <li><a href="#id3">Basic tests.</a></li> <li><a href="#id4">Testing the Controls.</a></li> <li><a href="#id5">Testing the Output.</a></li> <li><a href="#id6">Further Tests.</a></li> </ul> <p>Each section will be an introduction to an idea in testing shiny apps with Selenium, and point to more detailed explanation in other vignettes.</p> <h2><a id="id1">Some thoughts on testing.</a></h2> <h3>Why test?</h3> <p>When faced with testing for the first time the natural reaction is to think what now? what do i test? how much/many tests do I write? Tests need to do something useful to survive. Automated tests should help the team to make the next move by providing justified confidence a bug has been fixed, confirming refactored code still works as intended, or demonstrating that new features have been successfully implemented. There should be sufficient tests - neither more nor less: more increase the support burden, fewer leave us open to unpleasant surprises in production.</p> <p>One way to create our tests is to take the view of the user. What does the user want to do? They want to see this particular graph of a given data set. How do they do that? They select various options and input various choices. From this list of actions we can create an outline of our code for the test. </p> <p>For each method, we need to work out how to implement it in code. How could an automated test select the sliderInput bar? Do alternative ways exist? An understanding of HTML, CSS, and JavaScript will help you if you plan to use browser automation tools. All the visible elements of a web application are reflected in the Document Object Model (DOM) in HTML, and they can be addressed in various ways. Some simple examples of interacting with the DOM using <code>RSelenium</code> are given in the <code>Rselenium-basic</code> vignette.</p> <h3>Vary the tests.</h3> <p>Having static tests can lead to problems. Introducing variance into the tests can help pick up unexpected errors. This can be achieved by introducing an element of randomness into automatic inputs or randomizing order of selection etc.</p> <h3>Vary the browsers/OS.</h3> <p>It can help to test against a variety of browsers and operating systems. <code>RSelenium</code> can interact with services like <a href="http://saucelabs.com/">sauceLabs</a>. <code>sauceLabs</code> allows one to choose the browser or operating system or the version of the selenium server to use. You can test with iOS/Android/Windows/Mac/Linux and browsers like firefox/chrome/ie/opera/safari. This can be very useful to test how your app works on a range of platforms. More detailed information and examples can be seen on the sauceLabs vignette.</p> <h3>Record the tests.</h3> <p>RSelenium has the ability to take screenshots of the browser at a particular point in time. On failure of a test a screenshot can be useful to understand what happened. If you interface RSelenium with <code>sauceLabs</code> you get screenshots and videos automatically. See the sauceLabs vignette for further details.</p> <h3>Test for fixes.</h3> <p>Lots of bugs are discovered by means other than automated testing - they might be reported by users, for example. Once these bugs are fixed, the fixes must be tested. The tests must establish whether the problem has been fixed and, where practical, show that the root cause has been addressed. Since we want to make sure the bug doesn&#39;t resurface unnoticed in future releases, having automated tests for the bug seems sensible.</p> <h2><a id="id2">The shiny test app.</a></h2> <h3>Introduction</h3> <p>The shiny test app is composed of various widgets from the shiny package (0.8.0.99 at time of writing). We have also included the <code>ggplot2</code> library as output for one of the charts adapted from a discussion on <a href="http://stackoverflow.com/questions/11687739/two-legends-based-on-different-datasets-with-ggplot2">stackoverflow</a>. The app includes examples of some of the controls included with the <code>shiny</code> package namely <code>selectInput</code>, <code>numericInput</code>, <code>dateRangeInput</code> and a <code>sliderInput</code>. These controls are used to produce output rendered using <code>renderPrint</code>, <code>renderPlot(base)</code> , <code>renderPlot(ggplot2)</code> and <code>renderDataTable</code>. The app can be viewed if you have <code>shiny</code> installed. </p> <pre><code>require(shiny) runApp(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp&quot;), port = 6012) </code></pre> <p>Alternatively there is a version of the app running on the <code>RStudio</code> test server <code>spark</code> at <a href="http://spark.rstudio.com/johnharrison/shinytestapp/"><code>http://spark.rstudio.com/johnharrison/shinytestapp/</code></a>.</p> <p>An image of the app using <code>RSelenium</code> on a windows 8.1 machine running firefox 26.0</p> <h6 align = center>shinytestapp on win 8.1 firefox 26.0</h6> <p><img src="https://dl.dropboxusercontent.com/u/38391057/RSelenium/shinytesting/shinytestapp.png" title = "shinytestapp on win 8.1 firefox 26.0"/></p> <p>The image was generated using <code>RSelenium</code> and the following code.</p> <pre><code>user &lt;- &quot;rselenium0&quot; pass &lt;- &quot;***************************&quot; port &lt;- 80 ip &lt;- paste0(user, &#39;:&#39;, pass, &quot;@ondemand.saucelabs.com&quot;) browser &lt;- &quot;firefox&quot; version &lt;- &quot;26&quot; platform &lt;- &quot;Windows 8.1&quot; extraCapabilities &lt;- list(name = &quot;shinytestapp screenshot&quot;, username = user, accessKey = pass) remDr &lt;- remoteDriver$new(remoteServerAddr = ip, port = port, browserName = browser , version = version, platform = platform , extraCapabilities = extraCapabilities) remDr$open() remDr$navigate(&quot;http://spark.rstudio.com/johnharrison/shinytestapp/&quot;) webElems &lt;- remDr$findElements(&quot;css selector&quot;, &quot;#ctrlSelect input&quot;) lapply(webElems, function(x){x$clickElement()}) scr &lt;- remDr$screenshot(display = TRUE) </code></pre> <h3>Observations</h3> <p>From the screenshot we retrieved from the remote Driver there are some interesting observations to make. Note that the <code>selectInput</code> and <code>numericInput</code> boxes are sticking out. This is occuring because the sidePanel is given a bootstrap span of 3. This is however fluid. The resolution on the remote machine is low so the pixel count on the span 3 is also low. On a local machine with high resolution (Nothing amazing just a laptop) we did not observe the <code>selectInput</code> and <code>numericInput</code> boxes sticking out. </p> <p>We could have run with a higher resolution by passing the additional <code>screen-resolution</code> parameter to <code>sauceLabs</code>. </p> <pre><code>extraCapabilities &lt;- list(name = &quot;shinytestapp screenshot&quot;, username = user , accessKey = pass, &quot;screen-resolution&quot; = &quot;1280x1024&quot;) </code></pre> <h6 align = center>shinytestapp on win 8.1 firefox 26.0 res 1280x1024</h6> <p><img src="https://dl.dropboxusercontent.com/u/38391057/RSelenium/shinytesting/STA-highres.png" title = "shinytestapp on win 8.1 firefox 26.0 res 1280x1024"/></p> <p>We can see things look a bit better but the <code>data-table</code> search box is a bit compacted.</p> <h3>Inputs and Outputs</h3> <p>The app is designed to show testing of the basic shiny components. It is a bit contrived so testing it may not be as natural as testing a live working app. The outputs (charts and tables) are designed to sit side by side if possible with a maximum of 2 on a &ldquo;row&rdquo; then drop down to the next &ldquo;row&rdquo;. We can test to see if this is happening by checking the posistionof elements. We will investigate this later. </p> <h2><a id="id3">Basic tests.</a></h2> <h3>Basic Functionality</h3> <p>The first test we will look at implementing will be basic connection to the app. Typically we would make a request for the page and then observe what status code was returned. Selenium doesnt currently give the html status code of a navigation request so instead we will check if the title of the web page is correct. Our <code>Shiny Test App</code> has a title of &ldquo;Shiny Test App&rdquo; so we will check for this.</p> <p>We create a <code>test/</code> directory in our <code>Shiny Test App</code> folder. The first set of tests will be basic so we create a file <code>test-basic.r</code>. In this file we have the following code to start with:</p> <pre><code>context(&quot;basic&quot;) library(RSelenium) library(testthat) remDr &lt;- remoteDriver() remDr$open(silent = TRUE) appURL &lt;- &quot;http://127.0.0.1:6012&quot; test_that(&quot;can connect to app&quot;, { remDr$navigate(appURL) appTitle &lt;- remDr$getTitle()[[1]] expect_equal(appTitle, &quot;Shiny Test App&quot;) }) remDr$close() </code></pre> <p>We have a context of &ldquo;basic&rdquo; for the tests in this file. The test &ldquo;can connect to app&rdquo; simply navigates to the app URL and attempts to get the page title. If the page title is &ldquo;Shiny Test App&rdquo; the test is deemed successful. For testing purposes we assume the app is running locally. The easiest way to do this is open a second R session and issue the command: <code>runApp(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp&quot;), port = 6012)</code>. The second R session will listen for connection on port 6012 and return the <code>Shiny Test App</code>. If we ran this basic test we would expect the following output:</p> <pre><code>&gt; test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;), filter = &#39;basic&#39;, reporter = &quot;Tap&quot;) [1] &quot;Connecting to remote server&quot; 1..1 # Context basic ok 1 can connect to app </code></pre> <p>So running the test we observe that we can successfully &ldquo;connect&rdquo; to the <code>Shiny Test App</code>. What other functionality can we add to our &ldquo;basic&rdquo; test context. We can check that the controls and the tabs are present. We can add these tests to our <code>test-basic.r</code> file. </p> <pre><code>test_that(&quot;controls are present&quot;, { webElems &lt;- remDr$findElements(&quot;css selector&quot;, &quot;#ctrlSelect label&quot;) appCtrlLabels &lt;- sapply(webElems, function(x){x$getElementText()}) expect_equal(appCtrlLabels[[1]], &quot;Select controls required:&quot;) expect_equal(appCtrlLabels[[2]], &quot;selectInput&quot;) expect_equal(appCtrlLabels[[3]], &quot;numericInput&quot;) expect_equal(appCtrlLabels[[4]], &quot;dateRangeInput&quot;) expect_equal(appCtrlLabels[[5]], &quot;sliderInput&quot;) }) test_that(&quot;tabs are present&quot;, { webElems &lt;- remDr$findElements(&quot;css selector&quot;, &quot;.nav a&quot;) appTabLabels &lt;- sapply(webElems, function(x){x$getElementText()}) expect_equal(appTabLabels[[1]], &quot;Plots&quot;) expect_equal(appTabLabels[[2]], &quot;About&quot;) }) </code></pre> <p>When we rerun our basic test we should hopefully now see that it is checking for the prescence of the controls and the tabs.</p> <pre><code>&gt; test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;), filter = &#39;basic&#39;, reporter = &quot;Tap&quot;) [1] &quot;Connecting to remote server&quot; 1..8 # Context basic ok 1 can connect to app ok 2 controls are present ok 3 controls are present ok 4 controls are present ok 5 controls are present ok 6 controls are present ok 7 tabs are present ok 8 tabs are present </code></pre> <p>That concludes our basic test of the <code>Shiny Test App</code> functionality. Next we look at testing the input controls.</p> <h2><a id="id4">Testing the Controls</a></h2> <p>Our first test of the controls will be the functioning of the checkbox. We open a new file in the test directory of our <code>Shiny Test App</code> and give it the name <code>test-checkbox.r</code>. We also give it a context of <code>controls</code>.</p> <pre><code>context(&quot;controls&quot;) library(RSelenium) library(testthat) remDr &lt;- remoteDriver() remDr$open(silent = TRUE) sysDetails &lt;- remDr$getStatus() browser &lt;- remDr$sessionInfo$browserName appURL &lt;- &quot;http://127.0.0.1:6012&quot; test_that(&quot;can select/deselect checkbox 1&quot;, { remDr$navigate(appURL) webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#ctrlSelect1&quot;) initState &lt;- webElem$isElementSelected()[[1]] # check if we can select/deselect if(browser == &quot;internet explorer&quot;){ webElem$sendKeysToElement(list(key = &quot;space&quot;)) }else{ webElem$clickElement() } changeState &lt;- webElem$isElementSelected()[[1]] expect_is(initState, &quot;logical&quot;) expect_is(changeState, &quot;logical&quot;) expect_false(initState == changeState) }) remDr$close() </code></pre> <p>In this case I am informed there maybe issues with <code>Internet Explorer</code>. Usually one would select the element for the checkbox and click it. In the case of <code>Internet Explorer</code> it maybe necessary to pass a <code>space</code> key to the element instead. Otherwise the test is straightforward. We check the initial state of the checkbox. We click the checkbox or send a keypress of space to it. We check the changed state of the checkbox. If the initial state is different to the changed state the test is deemed a success. For good measure we also check that the initial and changed states are of class &ldquo;logical&rdquo;. We add code for the other 3 checkboxes. We can check our test as follows:</p> <pre><code>&gt; test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;), reporter = &quot;Tap&quot;, filter = &quot;checkbox&quot;) [1] &quot;Connecting to remote server&quot; 1..12 # Context controls ok 1 can select/deselect checkbox 1 ok 2 can select/deselect checkbox 1 ok 3 can select/deselect checkbox 1 ok 4 can select/deselect checkbox 2 ok 5 can select/deselect checkbox 2 ok 6 can select/deselect checkbox 2 ok 7 can select/deselect checkbox 3 ok 8 can select/deselect checkbox 3 ok 9 can select/deselect checkbox 3 ok 10 can select/deselect checkbox 4 ok 11 can select/deselect checkbox 4 ok 12 can select/deselect checkbox 4 </code></pre> <p>We filter here on &ldquo;checkbox&rdquo; to only select this test file to run. If you watch the test running it will filter through the checkbox control checking each checkbox is functioning. The <code>checkboxGroupInput</code> drives the required controls which has id <code>reqcontrols</code>. Each of these controls is one of the building blocks of shiny and we will add a test for each.</p> <h3>Testing the selectInput</h3> <p>We write a simple test for the <code>selectInput</code>. It tests the options presented and the label of the control. We isolate the code in a seperate file <code>test-selectinput.r</code> in the test folder of our <code>Shiny Test App</code>. It also then selects an element from the options at random. It is tested whether the output changes or not.</p> <pre><code>test_that(&quot;selectInput dataSet correct&quot;, { remDr$navigate(appURL) webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#ctrlSelect1&quot;) initState &lt;- webElem$isElementSelected()[[1]] if(!initState){ # select the checkbox if(browser == &quot;internet explorer&quot;){ webElem$sendKeysToElement(list(key = &quot;space&quot;)) }else{ webElem$clickElement() } } webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#reqcontrols #dataset&quot;) # check the available datasets childElems &lt;- webElem$findChildElements(&quot;css selector&quot;, &quot;[value]&quot;) appDataSets &lt;- sapply(childElems, function(x){x$getElementAttribute(&quot;value&quot;)}) expect_true(all(c(&quot;rock&quot;, &quot;pressure&quot;, &quot;cars&quot;) %in% appDataSets)) }) test_that(&quot;selectInput label correct&quot;, { webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#reqcontrols label[for = &#39;dataset&#39;]&quot;) expect_output(webElem$getElementText()[[1]], &quot;Choose a dataset:&quot;) } ) test_that(&quot;selectInput selection invokes change&quot;, { webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#reqcontrols #dataset&quot;) childElems &lt;- webElem$findChildElements(&quot;css selector&quot;, &quot;[value]&quot;) ceState &lt;- sapply(childElems, function(x){x$isElementSelected()}) newState &lt;- sample(seq_along(ceState)[!unlist(ceState)], 1) outElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#summary&quot;) initOutput &lt;- outElem$getElementText()[[1]] # change dataset childElems[[newState]]$clickElement() outElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#summary&quot;) changeOutput &lt;- outElem$getElementText()[[1]] expect_false(initOutput == changeOutput) } ) </code></pre> <p>Running the <code>selectInput</code> test we get:</p> <pre><code>&gt; test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;), reporter = &quot;Tap&quot;, filter = &quot;selectinput&quot;) [1] &quot;Connecting to remote server&quot; 1..3 # Context controls ok 1 selectInput dataSet correct ok 2 selectInput label correct ok 3 selectInput selection invokes change </code></pre> <p>Note we set <code>remDr$setImplicitWaitTimeout(3000)</code> in this test so that we get a 3 second limit to find an element. </p> <h3>Testing the numericInput</h3> <p>The ideas behind testing the numericInput are similar to testing the selectInput. We test the label. We then test a random value between the allowable limits of the numericInput and check that the output changes. Finally a character string &ldquo;test&rdquo; is sent to the element and the appropriate error message on the output is checked. The final test can be adjusted to suit whatever bespoke error display etc is in your app. The test code is in the tests folder of the <code>Shiny Test App</code> in a file named <code>test-numericinput.r</code>. Again <code>remDr$setImplicitWaitTimeout(3000)</code> is called to give some leeway for element loading. Some commented out code indicates other methods one could deal with checking for element existence. Additional detail on timing races in Selenium can be found <a href="http://www.bizalgo.com/2012/01/14/timing-races-selenium-2-implicit-waits-explicit-waits/">here</a>.</p> <h3>Testing the dateRangeInput</h3> <p>The test on the dateRangeInput compose of two tests. We test the label and we test the two input dates. We choose two random dates from the set of allowable dates. The output is tested for change after the two dates ave been set. <code>remDr$setImplicitWaitTimeout(3000)</code> is set in the test to allow for race conditions on elements. The test code is in the tests folder of the <code>Shiny Test App</code> in a file named <code>test-daterangeinput.r</code>.</p> <h3>Testing the sliderInput</h3> <p>For the sliderInput we test the label and we test changing the controls. The test code is in the tests folder of the <code>Shiny Test App</code> in a file named <code>test-sliderinput.r</code>. The label is tested in a similar fashion as the other controls. The second test needs a bit of explaining. There are a number of ways we could interact with the slider control to change its values. Some of the easiest ways would be to execute javascript with <code>Shiny.onInputChange(&quot;range&quot;, [2000, 10000])</code> or <code>Shiny.shinyapp.sendInput({range: [6222, 9333]})</code>. Both these methods would currently work. The Shiny server side would get the new values however the UI would show no change. The underlying sliderInput control is a <code>jslider</code>. Normally one can interact with the <code>jslider</code> thru calls similar to <code>$(&quot;.selector&quot;).slider(&quot;value&quot;, p1, p2)</code> as outlined <a href="http://egorkhmelev.github.io/jslider/">here</a>. We will use mouse movements and the <code>buttondown</code> <code>buttonup</code> methods of the remoteDriver class. <strong>Note that one may have problems forming the test in this manner, see for example <a href="http://stackoverflow.com/questions/19922578/understanding-of-cannot-perform-native-interaction-could-not-load-native-event">here</a></strong>. However it is useful to illustrate mouse and keyboard interactions in <code>RSelenium</code>.</p> <p>We get the attributes of the slider initially. We then get the dimension of the slider</p> <pre><code>webElem &lt;- remDr$findElement(&quot;css selector&quot;, &quot;#reqcontrols input#range + .jslider&quot;) sliderDim &lt;- webElem$getElementSize() </code></pre> <p>This gives us the pixel width of the slider as it currently stands. This will be different across machines. We generate some random values for the two slider points and then we calculate roughly how many pixels we need to move the sliders.</p> <pre><code>remDr$mouseMoveToLocation(webElement = webElems[[x]]) remDr$buttondown() remDr$mouseMoveToLocation(x = as.integer(pxToMoveSldr[x]), y = -1L)#, webElement = webElems[[x]]) remDr$buttonup() </code></pre> <p>The above code moves to the slider element. Pushes the left button down. Moves the mouse on the x axis in the direction calculated then releases the left mouse button. The output of the related data-table before and after the change is recorded and the test should result in the before and after not being equal.</p> <p>It is interesting to note that during initial writing of this vignette a new version of firefox 27.0.1 was released. As expected native events did not work under version 2.39 of selenium server and this updated version of firefox. Subsequently our test as formulated above would fail. There is an option to pass a list <code>rsel.opt</code> for use with some of the tests. Using this we can set <code>nativeEvents = FALSE</code> and the test above will pass again. When your tests fail it is not necessarily bad. This failure indicates a problem with your test setup rather then your app however.</p> <pre><code>testsel &lt;- test_env() with(testsel, rsel.opt &lt;- list(nativeEvents = FALSE)) test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;), reporter = &quot;Tap&quot;, filter = &quot;slider&quot;, env = testsel) </code></pre> <h2><a id="id5">Testing the Output.</a></h2> <p>Finally for this simple example we will look at testing the output. The test code is in the tests folder of the <code>Shiny Test App</code> in a file named <code>test-output.r</code>. The outputs should line up side by side with a maximum of 2 on a line. We can check the position of the outputs. Our first test will check whether the four outputs line up in a grid. This test will fail on low resolution setups which we will observe latter. We can check the headers on the outputs. The two chart plots are base64 encoded images which we can check in the HTML source. We can check the headers on the outputs. Finally we can check the controls on the datatable.</p> <p>The first test use the <code>getElementLocation</code> method of the <code>webElement</code> class to find the location in pixels of the output objects.</p> <pre><code>webElems &lt;- remDr$findElements(&quot;css selector&quot;, &quot;#reqplots .span5&quot;) out &lt;- sapply(webElems, function(x){x$getElementLocation()}) </code></pre> <p>The 1st and 2nd and the 3rd and 4th objects should share rows. The 1st and 3rd and the 2nd and 4th should share a column. This test will fail as the resolution of the app decreases and the output objects get compacted. The second test checks output labels in a similar fashion to other test. The third test checks whether the chart output are base 64 encoded png. The final test selects the data-table output and randomly selects a column from carat or price. It then checks whether the ordering functions when the column header is clicked.</p> <p>Finally running all tests with a &ldquo;summary&rdquo; reporter we would hope to get:</p> <pre><code>&gt; test_dir(paste0(find.package(&quot;RSelenium&quot;), &quot;/apps/shinytestapp/tests/&quot;)) basic : [1] &quot;Connecting to remote server&quot; ........ controls : [1] &quot;Connecting to remote server&quot; ............ controls : [1] &quot;Connecting to remote server&quot; .. controls : [1] &quot;Connecting to remote server&quot; ... outputs : [1] &quot;Connecting to remote server&quot; ....... controls : [1] &quot;Connecting to remote server&quot; ... controls : [1] &quot;Connecting to remote server&quot; .. </code></pre> <h2><a id="id6">Further Tests.</a></h2> <ul> <li>Test across multiple browsers and OS. See the saucelabs testing vignette</li> <li>Longitudinal type test. Record access times for various components of your app across time. See the RBMproxy testing vignette.</li> <li>Analysis current page load times. See the RBMproxy vignette</li> </ul> <p>