JetBrains plugin manual test cases

Overview

All these test scripts are written in a way so that humans can perform it, but also an E2E testing tool can run it eventually.

Each of these test cases should be run either right after starting the IDE and opening a project, or after closing and reopening a project, to make sure we have a clean state.

Test case format anatomy

The numbered items are the steps to perform in order.
The indented bullets are assertions.

Performing a test manually

If a red error notification appears at the bottom right of the IDE, that means we have a problem.

Same if any of the assertions fail.

If there is a problem, we should either investigate right away (typically, if a developer does the manual testing), or create an issue with the error message / specifying which test case failed, and any additional helpful context.

Test cases

Preparation

Perforce testing is slightly tricky because we don’t currently have a lot of Perforce code hosts added to any of our Sourcegraph instances. Try this:

Use https://cse-k8s.sgdev.org/ as your Sourcegraph URL
Generate an access token for yourself for CSE-K8S (admin user in 1Password), set that
Set up a Perforce client
Set perforce.sgdev.org:1666 for the port. User is “admin”, password is in 1Password.
Make sure your connection work, and actually connect in P4V (connection didn’t work without this step in IntelliJ for me)
Create a workspace for yourself (delete an old one if you bump into the 20-workspace limit), and include the test-large-depot depot
In the JetBrains plugin settings, set the replacement string to perforce.sgdev.org:,perforce.beatrix.com:app/,.perforce,/patch/core.perforce.
At this point, your files should open properly in your browser.

Search Selection and Open/Copy features

Open a project with a file that’s under Git version control
Right click on editor | Sourcegraph | Search Selection on Sourcegraph Web
- Make sure it searches the current selection in all repos
Right click on editor | Sourcegraph | Search Selection in Repository on Sourcegraph Web
- Make sure it searches the current selection in the current repo
Right click on editor | Sourcegraph | Open Selection in Sourcegraph Web
- Make sure it opens the right file on the web
Copy Sourcegraph File Link
- Make sure it copies the right URL (should be good if the previous point worked because they share the logic.)
Repeat it with a Perforce repo.

“Open Revision Diff in Sourcegraph Web”

Open a project that has files under Git version control
Open Version Control (⌘9) | Log
Right click on a commit and choose Open Revision Diff in Sourcegraph Web
Make sure the right page opens on the Sourcegraph instance set up in the plugin settings
(Would be cool to also test this each time with a project that has files from multiple repositories.)
Repeat the process with a project that has files under Perforce version control.

Find with Sourcegraph

Why: To make sure the popup is discoverable and generally works

Press the shortcut for the popup
- The popup should become active
- The popup should be in the “loading” state
Wait until the “loading” state is gone
- It should not take more than ~five seconds
Close the popup with ESC
- The popup should become hidden
Open “Find Action” (⌘⇧A) and search for “find with sourcegraph” and select the first result
- The popup should become active
- The popup should not be in the “loading” state
Click the header of the IDE main window
- The popup should become hidden
In the main menu, choose Edit | Find | Find on Sourcegraph…
- The popup should become active

Browser doesn’t disappear after a few opens

Why: After opening and closing the "Find on Sourcegraph" popup four-five times on Mac, the browser disappears if circumstances are not right. This bug should be fixed, but it may reappear if we mess something up.

Open popup with the shortcut
- The popup should become active
Wait until the “loading” state is gone
Close the popup with ESC
Open popup with the shortcut
- The popup should become active, and the browser should be visible
Repeat 3–4 at least five times.

Why: To make sure that the most basic functionality of the popup works

Open the popup with the shortcut
Wait for the browser content to load
Type “repo:^github\.com/sourcegraph/sourcegraph file:index.ts”
- It should type into the input box
- The content should be exactly the same as what was typed
Press Enter
Wait for the search results and preview to load
- The result list and the preview box should populate
Press ↓ key
Wait for the search results and preview to load
- The preview should be different from the previous one
Press ⌥Enter
- The popup should close
- A new editor tab should open
- The content of the new tab should be the same as the last preview
Open the popup with the shortcut
Press ⌘A, type “repo:^github\.com/sourcegraph/sourcegraph” and press Enter
Wait for the search results to load
- The result list should update
- The preview state should be “No preview available”
Press ⌥Enter
- The popup should close
- A browser should open (if automated: an “open browser” trigger should be sent)

Server connection success/failure is recognized

This test needs a valid access token that can be generated at https://sourcegraph.com/users/{username}/settings/tokens

Open Settings with ⌘,
Go to Tools | Sourcegraph
Set the URL to “https://sourcegraph.com”
Set the access token to a valid one
- Press Enter to save settings
Open the popup with the shortcut
Wait for the browser content to load.
- The search box should be visible
Close the popup with ESC
Open Settings with ⌘,
Type “TEST” after the end of the valid access token.
Press Enter to save settings
Open the popup with the shortcut
- The search box should not be visible, an error message should appear
- The preview should be saying “No preview available”
Close the popup with ESC
Open Settings with ⌘,
Remove “TEST” to again have the valid access token.
Press Enter to save settings
Open the popup with the shortcut
- The search box should be visible again.