Fix touch events HiDPI-scaling [TODO: rebase on dev]

Touch events were HiDPI-scaled twice:
 - once because the position (provided as floats between 0 and 1) were
   converted in pixels using the drawable size (not the window size)
 - once due to screen_convert_to_frame_coords()

One possible fix could be to compute the position in pixels from the
window size instead, but this would unnecessarily round the event
position to the nearest window coordinates (instead of drawable
coordinates).

Instead, expose two separate functions to convert to frame coordinates
from either window or drawable coordinates.

Fixes #1536 <https://github.com/Genymobile/scrcpy/issues/1536>
Refs #15 <https://github.com/Genymobile/scrcpy/issues/15>
Refs e40532a376

README.md

@@ -301,7 +301,7 @@ ssh -CN -L5037:localhost:5037 -L27183:localhost:27183 your_remote_computer
 From another terminal:
 
 ```bash
-scrcpy --force-adb-forwrad
+scrcpy --force-adb-forward
 ```
 
 
@@ -550,43 +550,40 @@ scrcpy --push-target /sdcard/foo/bar/
 
 ### Audio forwarding
 
-Audio is not forwarded by _scrcpy_. Use [USBaudio] (Linux-only).
+Audio is not forwarded by _scrcpy_. Use [sndcpy].
 
 Also see [issue #14].
 
-[USBaudio]: https://github.com/rom1v/usbaudio
+[sndcpy]: https://github.com/rom1v/sndcpy
 [issue #14]: https://github.com/Genymobile/scrcpy/issues/14
 
 
 ## Shortcuts
 
-`LCtrl` is the left `Ctrl` key (the right `Ctrl` key is forwarded to the
-device).
-
  | Action                                      |   Shortcut                    |   Shortcut (macOS)
  | ------------------------------------------- |:----------------------------- |:-----------------------------
- | Switch fullscreen mode                      | `LCtrl`+`f`                    | `Cmd`+`f`
+ | Switch fullscreen mode                      | `Ctrl`+`f`                    | `Cmd`+`f`
- | Rotate display left                         | `LCtrl`+`←` _(left)_           | `Cmd`+`←` _(left)_
+ | Rotate display left                         | `Ctrl`+`←` _(left)_           | `Cmd`+`←` _(left)_
- | Rotate display right                        | `LCtrl`+`→` _(right)_          | `Cmd`+`→` _(right)_
+ | Rotate display right                        | `Ctrl`+`→` _(right)_          | `Cmd`+`→` _(right)_
- | Resize window to 1:1 (pixel-perfect)        | `LCtrl`+`g`                    | `Cmd`+`g`
+ | Resize window to 1:1 (pixel-perfect)        | `Ctrl`+`g`                    | `Cmd`+`g`
- | Resize window to remove black borders       | `LCtrl`+`x` \| _Double-click¹_ | `Cmd`+`x`  \| _Double-click¹_
+ | Resize window to remove black borders       | `Ctrl`+`x` \| _Double-click¹_ | `Cmd`+`x`  \| _Double-click¹_
- | Click on `HOME`                             | `LCtrl`+`h` \| _Middle-click_  | `Ctrl`+`h` \| _Middle-click_
+ | Click on `HOME`                             | `Ctrl`+`h` \| _Middle-click_  | `Ctrl`+`h` \| _Middle-click_
- | Click on `BACK`                             | `LCtrl`+`b` \| _Right-click²_  | `Cmd`+`b`  \| _Right-click²_
+ | Click on `BACK`                             | `Ctrl`+`b` \| _Right-click²_  | `Cmd`+`b`  \| _Right-click²_
- | Click on `APP_SWITCH`                       | `LCtrl`+`s`                    | `Cmd`+`s`
+ | Click on `APP_SWITCH`                       | `Ctrl`+`s`                    | `Cmd`+`s`
- | Click on `MENU`                             | `LCtrl`+`m`                    | `Ctrl`+`m`
+ | Click on `MENU`                             | `Ctrl`+`m`                    | `Ctrl`+`m`
- | Click on `VOLUME_UP`                        | `LCtrl`+`↑` _(up)_             | `Cmd`+`↑` _(up)_
+ | Click on `VOLUME_UP`                        | `Ctrl`+`↑` _(up)_             | `Cmd`+`↑` _(up)_
- | Click on `VOLUME_DOWN`                      | `LCtrl`+`↓` _(down)_           | `Cmd`+`↓` _(down)_
+ | Click on `VOLUME_DOWN`                      | `Ctrl`+`↓` _(down)_           | `Cmd`+`↓` _(down)_
- | Click on `POWER`                            | `LCtrl`+`p`                    | `Cmd`+`p`
+ | Click on `POWER`                            | `Ctrl`+`p`                    | `Cmd`+`p`
  | Power on                                    | _Right-click²_                | _Right-click²_
- | Turn device screen off (keep mirroring)     | `LCtrl`+`o`                    | `Cmd`+`o`
+ | Turn device screen off (keep mirroring)     | `Ctrl`+`o`                    | `Cmd`+`o`
- | Turn device screen on                       | `LCtrl`+`Shift`+`o`            | `Cmd`+`Shift`+`o`
+ | Turn device screen on                       | `Ctrl`+`Shift`+`o`            | `Cmd`+`Shift`+`o`
- | Rotate device screen                        | `LCtrl`+`r`                    | `Cmd`+`r`
+ | Rotate device screen                        | `Ctrl`+`r`                    | `Cmd`+`r`
- | Expand notification panel                   | `LCtrl`+`n`                    | `Cmd`+`n`
+ | Expand notification panel                   | `Ctrl`+`n`                    | `Cmd`+`n`
- | Collapse notification panel                 | `LCtrl`+`Shift`+`n`            | `Cmd`+`Shift`+`n`
+ | Collapse notification panel                 | `Ctrl`+`Shift`+`n`            | `Cmd`+`Shift`+`n`
- | Copy device clipboard to computer           | `LCtrl`+`c`                    | `Cmd`+`c`
+ | Copy device clipboard to computer           | `Ctrl`+`c`                    | `Cmd`+`c`
- | Paste computer clipboard to device          | `LCtrl`+`v`                    | `Cmd`+`v`
+ | Paste computer clipboard to device          | `Ctrl`+`v`                    | `Cmd`+`v`
- | Copy computer clipboard to device and paste | `LCtrl`+`Shift`+`v`            | `Cmd`+`Shift`+`v`
+ | Copy computer clipboard to device and paste | `Ctrl`+`Shift`+`v`            | `Cmd`+`Shift`+`v`
- | Enable/disable FPS counter (on stdout)      | `LCtrl`+`i`                    | `Cmd`+`i`
+ | Enable/disable FPS counter (on stdout)      | `Ctrl`+`i`                    | `Cmd`+`i`
 
 _¹Double-click on black borders to remove them._  
 _²Right-click turns the screen on if it was off, presses BACK otherwise._

app/scrcpy.1

@@ -203,54 +203,52 @@ Default is 0 (automatic).\n
 
 .SH SHORTCUTS
 
-\fBLCtrl\fR is the left Ctrl key (the right Ctrl key is forwarded to the device).
-
 .TP
-.B LCtrl+f
+.B Ctrl+f
 Switch fullscreen mode
 
 .TP
-.B LCtrl+Left
+.B Ctrl+Left
 Rotate display left
 
 .TP
-.B LCtrl+Right
+.B Ctrl+Right
 Rotate display right
 
 .TP
-.B LCtrl+g
+.B Ctrl+g
 Resize window to 1:1 (pixel\-perfect)
 
 .TP
-.B LCtrl+x, Double\-click on black borders
+.B Ctrl+x, Double\-click on black borders
 Resize window to remove black borders
 
 .TP
-.B LCtrl+h, Home, Middle\-click
+.B Ctrl+h, Home, Middle\-click
 Click on HOME
 
 .TP
-.B LCtrl+b, Ctrl+Backspace, Right\-click (when screen is on)
+.B Ctrl+b, Ctrl+Backspace, Right\-click (when screen is on)
 Click on BACK
 
 .TP
-.B LCtrl+s
+.B Ctrl+s
 Click on APP_SWITCH
 
 .TP
-.B LCtrl+m
+.B Ctrl+m
 Click on MENU
 
 .TP
-.B LCtrl+Up
+.B Ctrl+Up
 Click on VOLUME_UP
 
 .TP
-.B LCtrl+Down
+.B Ctrl+Down
 Click on VOLUME_DOWN
 
 .TP
-.B LCtrl+p
+.B Ctrl+p
 Click on POWER (turn screen on/off)
 
 .TP
@@ -258,39 +256,39 @@ Click on POWER (turn screen on/off)
 Turn screen on
 
 .TP
-.B LCtrl+o
+.B Ctrl+o
 Turn device screen off (keep mirroring)
 
 .TP
-.B LCtrl+Shift+o
+.B Ctrl+Shift+o
 Turn device screen on
 
 .TP
-.B LCtrl+r
+.B Ctrl+r
 Rotate device screen
 
 .TP
-.B LCtrl+n
+.B Ctrl+n
 Expand notification panel
 
 .TP
-.B LCtrl+Shift+n
+.B Ctrl+Shift+n
 Collapse notification panel
 
 .TP
-.B LCtrl+c
+.B Ctrl+c
 Copy device clipboard to computer
 
 .TP
-.B LCtrl+v
+.B Ctrl+v
 Paste computer clipboard to device
 
 .TP
-.B LCtrl+Shift+v
+.B Ctrl+Shift+v
 Copy computer clipboard to device (and paste if the device runs Android >= 7)
 
 .TP
-.B LCtrl+i
+.B Ctrl+i
 Enable/disable FPS counter (print frames/second in logs)
 
 .TP

app/src/cli.c

@@ -15,7 +15,7 @@ scrcpy_print_usage(const char *arg0) {
 #ifdef __APPLE__
 # define CTRL_OR_CMD "Cmd"
 #else
-# define CTRL_OR_CMD "LCtrl"
+# define CTRL_OR_CMD "Ctrl"
 #endif
     fprintf(stderr,
         "Usage: %s [options]\n"
@@ -186,9 +186,6 @@ scrcpy_print_usage(const char *arg0) {
         "\n"
         "Shortcuts:\n"
         "\n"
-        "    LCtrl is the left Ctrl key (the right Ctrl key is forwarded to\n"
-        "    the device).\n"
-        "\n"
         "    " CTRL_OR_CMD "+f\n"
         "        Switch fullscreen mode\n"
         "\n"
@@ -205,7 +202,7 @@ scrcpy_print_usage(const char *arg0) {
         "    Double-click on black borders\n"
         "        Resize window to remove black borders\n"
         "\n"
-        "    LCtrl+h\n"
+        "    Ctrl+h\n"
         "    Middle-click\n"
         "        Click on HOME\n"
         "\n"
@@ -217,7 +214,7 @@ scrcpy_print_usage(const char *arg0) {
         "    " CTRL_OR_CMD "+s\n"
         "        Click on APP_SWITCH\n"
         "\n"
-        "    LCtrl+m\n"
+        "    Ctrl+m\n"
         "        Click on MENU\n"
         "\n"
         "    " CTRL_OR_CMD "+Up\n"

app/src/event_converter.c

@@ -92,11 +92,6 @@ convert_keycode(SDL_Keycode from, enum android_keycode *to, uint16_t mod,
         MAP(SDLK_LEFT,         AKEYCODE_DPAD_LEFT);
         MAP(SDLK_DOWN,         AKEYCODE_DPAD_DOWN);
         MAP(SDLK_UP,           AKEYCODE_DPAD_UP);
-        MAP(SDLK_RCTRL,        AKEYCODE_CTRL_RIGHT);
-        MAP(SDLK_LSHIFT,       AKEYCODE_SHIFT_LEFT);
-        MAP(SDLK_RSHIFT,       AKEYCODE_SHIFT_RIGHT);
-        MAP(SDLK_LALT,         AKEYCODE_ALT_LEFT);
-        MAP(SDLK_RALT,         AKEYCODE_ALT_RIGHT);
     }
 
     if (!(mod & (KMOD_NUM | KMOD_SHIFT))) {

app/src/input_manager.c

@@ -259,8 +259,8 @@ input_manager_process_key(struct input_manager *im,
     // control: indicates the state of the command-line option --no-control
     // ctrl: the Ctrl key
 
-    // Only capture Left-Ctrl, Right-Ctrl is forwarded to the device
+    bool ctrl = event->keysym.mod & (KMOD_LCTRL | KMOD_RCTRL);
-    bool ctrl = event->keysym.mod & KMOD_LCTRL;
+    bool alt = event->keysym.mod & (KMOD_LALT | KMOD_RALT);
     bool meta = event->keysym.mod & (KMOD_LGUI | KMOD_RGUI);
 
     // use Cmd on macOS, Ctrl on other platforms
@@ -275,6 +275,11 @@ input_manager_process_key(struct input_manager *im,
     bool cmd = ctrl; // && !meta, already guaranteed
 #endif
 
+    if (alt) {
+        // no shortcuts involve Alt, and it must not be forwarded to the device
+        return;
+    }
+
     struct controller *controller = im->controller;
 
     // capture all Ctrl events
@@ -422,7 +427,7 @@ convert_mouse_motion(const SDL_MouseMotionEvent *from, struct screen *screen,
     to->inject_touch_event.pointer_id = POINTER_ID_MOUSE;
     to->inject_touch_event.position.screen_size = screen->frame_size;
     to->inject_touch_event.position.point =
-        screen_convert_to_frame_coords(screen, from->x, from->y);
+        screen_convert_window_to_frame_coords(screen, from->x, from->y);
     to->inject_touch_event.pressure = 1.f;
     to->inject_touch_event.buttons = convert_mouse_buttons(from->state);
 
@@ -460,15 +465,15 @@ convert_touch(const SDL_TouchFingerEvent *from, struct screen *screen,
     to->inject_touch_event.pointer_id = from->fingerId;
     to->inject_touch_event.position.screen_size = screen->frame_size;
 
-    int ww;
+    int dw;
-    int wh;
+    int dh;
-    SDL_GL_GetDrawableSize(screen->window, &ww, &wh);
+    SDL_GL_GetDrawableSize(screen->window, &dw, &dh);
 
     // SDL touch event coordinates are normalized in the range [0; 1]
-    int32_t x = from->x * ww;
+    int32_t x = from->x * dw;
-    int32_t y = from->y * wh;
+    int32_t y = from->y * dh;
     to->inject_touch_event.position.point =
-        screen_convert_to_frame_coords(screen, x, y);
+        screen_convert_drawable_to_frame_coords(screen, x, y);
 
     to->inject_touch_event.pressure = from->pressure;
     to->inject_touch_event.buttons = 0;
@@ -498,7 +503,7 @@ convert_mouse_button(const SDL_MouseButtonEvent *from, struct screen *screen,
     to->inject_touch_event.pointer_id = POINTER_ID_MOUSE;
     to->inject_touch_event.position.screen_size = screen->frame_size;
     to->inject_touch_event.position.point =
-        screen_convert_to_frame_coords(screen, from->x, from->y);
+        screen_convert_window_to_frame_coords(screen, from->x, from->y);
     to->inject_touch_event.pressure = 1.f;
     to->inject_touch_event.buttons =
         convert_mouse_buttons(SDL_BUTTON(from->button));
@@ -563,7 +568,8 @@ convert_mouse_wheel(const SDL_MouseWheelEvent *from, struct screen *screen,
 
     struct position position = {
         .screen_size = screen->frame_size,
-        .point = screen_convert_to_frame_coords(screen, mouse_x, mouse_y),
+        .point = screen_convert_window_to_frame_coords(screen,
+                                                       mouse_x, mouse_y),
     };
 
     to->type = CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT;

app/src/screen.c

@@ -579,14 +579,14 @@ screen_handle_window_event(struct screen *screen,
 }
 
 struct point
-screen_convert_to_frame_coords(struct screen *screen, int32_t x, int32_t y) {
+screen_convert_drawable_to_frame_coords(struct screen *screen,
+                                        int32_t x, int32_t y) {
     unsigned rotation = screen->rotation;
     assert(rotation < 4);
 
     int32_t w = screen->content_size.width;
     int32_t h = screen->content_size.height;
 
-    screen_hidpi_scale_coords(screen, &x, &y);
 
     x = (int64_t) (x - screen->rect.x) * w / screen->rect.w;
     y = (int64_t) (y - screen->rect.y) * h / screen->rect.h;
@@ -615,6 +615,13 @@ screen_convert_to_frame_coords(struct screen *screen, int32_t x, int32_t y) {
     return result;
 }
 
+struct point
+screen_convert_window_to_frame_coords(struct screen *screen,
+                                      int32_t x, int32_t y) {
+    screen_hidpi_scale_coords(screen, &x, &y);
+    return screen_convert_drawable_to_frame_coords(screen, x, y);
+}
+
 void
 screen_hidpi_scale_coords(struct screen *screen, int32_t *x, int32_t *y) {
     // take the HiDPI scaling (dw/ww and dh/wh) into account

app/src/screen.h

@@ -126,7 +126,14 @@ screen_handle_window_event(struct screen *screen, const SDL_WindowEvent *event);
 // convert point from window coordinates to frame coordinates
 // x and y are expressed in pixels
 struct point
-screen_convert_to_frame_coords(struct screen *screen, int32_t x, int32_t y);
+screen_convert_window_to_frame_coords(struct screen *screen,
+                                      int32_t x, int32_t y);
+
+// convert point from drawable coordinates to frame coordinates
+// x and y are expressed in pixels
+struct point
+screen_convert_drawable_to_frame_coords(struct screen *screen,
+                                        int32_t x, int32_t y);
 
 // Convert coordinates from window to drawable.
 // Events are expressed in window coordinates, but content is expressed in

server/build_without_gradle.sh

@@ -42,6 +42,8 @@ echo "Generating java from aidl..."
 cd "$SERVER_DIR/src/main/aidl"
 "$ANDROID_HOME/build-tools/$BUILD_TOOLS/aidl" -o"$CLASSES_DIR" \
     android/view/IRotationWatcher.aidl
+"$ANDROID_HOME/build-tools/$BUILD_TOOLS/aidl" -o"$CLASSES_DIR" \
+    android/content/IOnPrimaryClipChangedListener.aidl
 
 echo "Compiling java sources..."
 cd ../java
@@ -55,6 +57,7 @@ cd "$CLASSES_DIR"
 "$ANDROID_HOME/build-tools/$BUILD_TOOLS/dx" --dex \
     --output "$BUILD_DIR/classes.dex" \
     android/view/*.class \
+    android/content/*.class \
     com/genymobile/scrcpy/*.class \
     com/genymobile/scrcpy/wrappers/*.class