flypig.co.uk

List items

Items from the current list are shown below.

Gecko

All items from January 2024

25 Jan 2024 : Day 149 #
It's my last dev diary before taking a 14 day break from gecko development today. I'm not convinced that I've made the right decision: there's a part of me that thinks I should forge on ahead and just make all of the things fit into the time I have available. But there's also the realist in me that says something has to give.

So there will be no dev diary tomorrow or until the 8th February, at which point I'll start up again. Ostensibly the reason is so that I can get my presentations together for FOSDEM. I want to do a decent job with the presentations. But I also have a lot going on at work right now. So it's necessary.

But there's still development to do today. Yesterday I set the build running after having added /browser/components/sessionstore to the embedlite/moz.build file. I was hoping this would result in SessionStore.jsm and its dependencies being added to omni.ja.

The package built fine. But after installing it on my device, the new files weren't to be found: they didn't make it into the archive. Worse than that, they've not even made it into the obj-build-mer-qt-x folder on my laptop. That means they're still not getting included in the build.

It rather makes sense as well. The LOCAL_INCLUDES value should, if I'm understanding correctly, list places where C++ headers might be found. This should affect JavaScript files at all.

So I've spent the day digging around in the build system trying to figure out what needs to be changed to get them where they need to be. I'm certain there's an easy answer, but I just can't seem to figure it out.

I thought about trying to override SessionStore.jsm as a component, but since it doesn't actually seem to be a component itself, this didn't work either.

So after much flapping around, I've decided just to patch out the functionality from the SessionStoreFunctions.jsm file. That doesn't feel like the right way to do this, but until someone suggests a better way (which I'm all open to!) this should at least be a pretty simple fix.

Let's see.

I've built a new version of gecko-dev with the changes applied, installed them on my phone and it's now time to run them. 
$ sailfish-browser 
[D] unknown:0 - Using Wayland-EGL
library "libGLESv2_adreno.so" not found
library "eglSubDriverAndroid.so" not found
greHome from GRE_HOME:/usr/bin
libxul.so is not found, in /usr/bin/libxul.so
Created LOG for EmbedLiteTrace
[D] onCompleted:105 - ViewPlaceholder requires a SilicaFlickable parent
Created LOG for EmbedLite
Created LOG for EmbedPrefs
Created LOG for EmbedLiteLayerManager
JavaScript error: chrome://embedlite/content/embedhelper.js, line 259:
    TypeError: sessionHistory is null
Call EmbedLiteApp::StopChildThread()
Redirecting call to abort() to mozalloc_abort
The log output is encouragingly quiet. There is one error stating that sessionHistory is null. I think this is unrelated to the SessionStore changes I've made, but it's still worth looking into this to fix it. Maybe this will be what fixes the Back and Forwards buttons?

What's clear is that the SessionStore errors have now gone, which is great. But fixing those errors sadly hasn't fixed the Back and Forwards buttons.

Let's look at this other error then. It's caused by the last line in this code block: 
      case "embedui:addhistory": {
        // aMessage.data contains: 1) list of 'links' loaded from DB, 2) current
        // 'index'.

        let docShell = content.docShell;
        let sessionHistory = docShell.QueryInterface(Ci.nsIWebNavigation).
            sessionHistory;
        let legacyHistory = sessionHistory.legacySHistory;
The penultimate line is essentially saying "View the docShell object as a WebNavigation object and access the sessionHistory value stored inside it".

So there are three potential reasons why this might be failing. First it could be that docShell no longer supports the WebNavigation interface. Second it could be that WebNavigation has changed so it no longer contains a sessionHitory value. Third it could be that the value is still there, but it's set to null.

From the nsDocShell class definition in nsDocShell.h it's clear that the interface is still supported: 
class nsDocShell final : public nsDocLoader,
                         public nsIDocShell,
                         public nsIWebNavigation,
                         public nsIBaseWindow,
                         public nsIRefreshURI,
                         public nsIWebProgressListener,
                         public nsIWebPageDescriptor,
                         public nsIAuthPromptProvider,
                         public nsILoadContext,
                         public nsINetworkInterceptController,
                         public nsIDeprecationWarner,
                         public mozilla::SupportsWeakPtr {
So let's check that WebNavigation interface, defined in the nsIWebNavigation.idl file. The field is still there in the interface definition: 
  /**
   * The session history object used by this web navigation instance. This
   * object will be a mozilla::dom::ChildSHistory object, but is returned as
   * nsISupports so it can be called from JS code.
   */
  [binaryname(SessionHistoryXPCOM)]
  readonly attribute nsISupports sessionHistory;
Although the interface is being accessed from nsDocShell, when we look at the code we can see that the history itself is coming from elsewhere: 
  mozilla::dom::ChildSHistory* GetSessionHistory() {
    return mBrowsingContext->GetChildSessionHistory();
  }
[...]
  RefPtr<mozilla::dom::BrowsingContext> mBrowsingContext;
This provides us with an opportunity, because it means we can place a breakpoint here to see what it's doing. 
$ gdb sailfish-browser
[...]
(gdb) b nsDocShell::GetSessionHistory
Breakpoint 1 at 0x7fbc7b37c4: nsDocShell::GetSessionHistory. (10 locations)
(gdb) b BrowsingContext::GetChildSessionHistory
Breakpoint 2 at 0x7fbc7b376c: file docshell/base/BrowsingContext.cpp, line 3314.
(gdb) c
Thread 10 "GeckoWorkerThre" hit Breakpoint 1, nsDocShell::GetSessionHistory
    (this=0x7f80aa9280)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:313
313     ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/RefPtr.h:
    No such file or directory.
(gdb) c
Continuing.

Thread 10 "GeckoWorkerThre" hit Breakpoint 2, mozilla::dom::BrowsingContext::
    GetChildSessionHistory (this=0x7f80c58e90)
    at docshell/base/BrowsingContext.cpp:3314
3314    ChildSHistory* BrowsingContext::GetChildSessionHistory() {
(gdb) b mChildSessionHistory
Function "mChildSessionHistory" not defined.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) p mChildSessionHistory
$1 = {mRawPtr = 0x0}
(gdb)
So this value is unset, but we'd expect it to be set as a consequence of a call to CreateChildSHistory(): 
void BrowsingContext::CreateChildSHistory() {
  MOZ_ASSERT(IsTop());
  MOZ_ASSERT(GetHasSessionHistory());
  MOZ_DIAGNOSTIC_ASSERT(!mChildSessionHistory);

  // Because session history is global in a browsing context tree, every process
  // that has access to a browsing context tree needs access to its session
  // history. That is why we create the ChildSHistory object in every process
  // where we have access to this browsing context (which is the top one).
  mChildSessionHistory = new ChildSHistory(this);

  // If the top browsing context (this one) is loaded in this process then we
  // also create the session history implementation for the child process.
  // This can be removed once session history is stored exclusively in the
  // parent process.
  mChildSessionHistory->SetIsInProcess(IsInProcess());
}
As I'm looking through this code in ESR 91 and ESR 78 I notice that the above method has changed: the call to SetIsInProcess() is new. I wonder if that will ultimately be related to why this isn't being set? I'm thinking that the location where the creation happens may be different.

There are indeed some differences. In ESR 91 it's called in BrowsingContext::CreateFromIPC() and BrowsingContext::Attach() whereas in ESR 78 it's called in BrowsingContext::Attach(). Both versions also have it being called from BrowsingContext::DidSet().

I should put some breakpoints on those methods to see which, if any, is being called. And I should do it for both versions.

See here's the result for ESR 91: 
$ gdb sailfish-browser 
[...]
(gdb) b BrowsingContext::CreateChildSHistory
Function "BrowsingContext::CreateChildSHistory" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (BrowsingContext::CreateChildSHistory) pending.
(gdb) r
[...]
The breakpoint is never hit and the creation never occurs. In contrast, on ESR 78 we get a hit before the first page has loaded: 
$ gdb sailfish-browser
[...]
(gdb) b BrowsingContext::CreateChildSHistory
Function "BrowsingContext::CreateChildSHistory" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (BrowsingContext::CreateChildSHistory) pending.
(gdb) r
[...]

Thread 8 "GeckoWorkerThre" hit Breakpoint 1, mozilla::dom::BrowsingContext::
    CreateChildSHistory (this=this@entry=0x7f889dc120)
    at obj-build-mer-qt-xr/dist/include/mozilla/cxxalloc.h:33
33      obj-build-mer-qt-xr/dist/include/mozilla/cxxalloc.h:
    No such file or directory.
(gdb) bt
#0  mozilla::dom::BrowsingContext::CreateChildSHistory
    (this=this@entry=0x7f889dc120)
    at obj-build-mer-qt-xr/dist/include/mozilla/cxxalloc.h:33
#1  0x0000007fbc7c933c in mozilla::dom::BrowsingContext::DidSet
    (aOldValue=<optimized out>, this=0x7f889dc120)
    at docshell/base/BrowsingContext.cpp:2356
#2  mozilla::dom::syncedcontext::Transaction<mozilla::dom::BrowsingContext>::
    Apply(mozilla::dom::BrowsingContext*)::{lambda(auto:1)#1}::operator()
    <std::integral_constant<unsigned long, 37ul> >(std::integral_constant
    <unsigned long, 37ul>) const (this=<optimized out>, this=<optimized out>,
    idx=...)
    at obj-build-mer-qt-xr/dist/include/mozilla/dom/SyncedContextInlines.h:137
[...]
#8  0x0000007fbc7cb484 in mozilla::dom::BrowsingContext::SetHasSessionHistory
    <bool> (this=this@entry=0x7f889dc120, 
    aValue=aValue@entry=@0x7fa6e1da57: true)
    at obj-build-mer-qt-xr/dist/include/mozilla/OperatorNewExtensions.h:47
#9  0x0000007fbc7cb54c in mozilla::dom::BrowsingContext::InitSessionHistory
    (this=0x7f889dc120)
    at docshell/base/BrowsingContext.cpp:2316
#10 0x0000007fbc7cb590 in mozilla::dom::BrowsingContext::InitSessionHistory
    (this=this@entry=0x7f889dc120)
    at obj-build-mer-qt-xr/dist/include/mozilla/dom/BrowsingContext.h:161
#11 0x0000007fbc901fac in nsWebBrowser::Create (aContainerWindow=
    <optimized out>, aParentWidget=<optimized out>, 
    aBrowsingContext=aBrowsingContext@entry=0x7f889dc120,
    aInitialWindowChild=aInitialWindowChild@entry=0x0)
    at toolkit/components/browser/nsWebBrowser.cpp:158
#12 0x0000007fbca950e8 in mozilla::embedlite::EmbedLiteViewChild::
    InitGeckoWindow (this=0x7f88bca840, parentId=0, parentBrowsingContext=0x0, 
    isPrivateWindow=<optimized out>, isDesktopMode=false)
    at obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:847
[...]
#33 0x0000007fb735e89c in ?? () from /lib64/libc.so.6
(gdb) c
Continuing.
[...]
This could well be our smoking gun. I need to check back through the backtrace to understand the process happening on ESR 78 and then establish why something similar isn't happening on ESR 91. Progress!

Immediately on examining the backtrace it's clear something odd is happening. The callee is BrowsingContext::DidSet() which is the only location where the call is made in both ESR 78 and ESR 91. So that does rather beg the question of why it's not getting called in ESR 91.

Digging back through the backtrace further, it eventually materialises that the difference is happening in nsWebBrowser::Create(). There's this bit of code in ESR 78 that looks like this: 
  // If the webbrowser is a content docshell item then we won't hear any
  // events from subframes. To solve that we install our own chrome event
  // handler that always gets called (even for subframes) for any bubbling
  // event.

  if (aBrowsingContext->IsTop()) {
    aBrowsingContext->InitSessionHistory();
  }

  NS_ENSURE_SUCCESS(docShellAsWin->Create(), nullptr);

  docShellTreeOwner->AddToWatcher();  // evil twin of Remove in SetDocShell(0)
  docShellTreeOwner->AddChromeListeners();
You can see the InitSessionHistory() call in there which eventually leads to the creation of our sessinHistory object. In ESR 91 that same bit of code looks like this: 
  // If the webbrowser is a content docshell item then we won't hear any
  // events from subframes. To solve that we install our own chrome event
  // handler that always gets called (even for subframes) for any bubbling
  // event.

  nsresult rv = docShell->InitWindow(nullptr, docShellParentWidget, 0, 0, 0, 0);
  if (NS_WARN_IF(NS_FAILED(rv))) {
    return nullptr;
  }

  docShellTreeOwner->AddToWatcher();  // evil twin of Remove in SetDocShell(0)
  docShellTreeOwner->AddChromeListeners();
Where has the InitSessionHistory() gone? It should be possible to find out using a bit of git log searching. Here we're following the rule of using git blame to find out about lines that have been added and git log -S to find out about lines that have been removed. 
$ git log -1 -S InitSessionHistory toolkit/components/browser/nsWebBrowser.cpp
commit 140a4164598e0c9ed537a377cf66ef668a7fbc25
Author: Randell Jesup <rjesup@wgate.com>
Date:   Mon Feb 1 22:57:12 2021 +0000

    Bug 1673617 - Create BrowsingContext::mChildSessionHistory more
    aggressively, r=nika
    
    Differential Revision: https://phabricator.services.mozilla.com/D100348
Just looking at this change, it removes the call to InitSessionHistory() in nsWebBrowser::Create() and moves it to nsFrameLoader::TryRemoteBrowserInternal. There are some related changes in the parent Bug 1673617, but looking through those it doesn't seem that they're anything we need to worry about.

Placing a breakpoint on nsFrameLoader::TryRemoteBrowserInternal() shows that it's not being called on ESR 91.

The interesting thing is that it appears that if EnsureRemoteBrowser() gets called in this bit of code: 
BrowsingContext* nsFrameLoader::GetBrowsingContext() {
  if (mNotifyingCrash) {
    if (mPendingBrowsingContext && mPendingBrowsingContext->EverAttached()) {
      return mPendingBrowsingContext;
    }
    return nullptr;
  }
  if (IsRemoteFrame()) {
    Unused << EnsureRemoteBrowser();
  } else if (mOwnerContent) {
    Unused << MaybeCreateDocShell();
  }
  return GetExtantBrowsingContext();
}
If the first path in the second condition is followed then the InitSessionHistory() would get called too. If this gets called by the InitSessionHistory() doesn't, it would imply that IsRemoteFrame() must be false. But if it is false then MaybeCreateDocShell() could get called, which also has a call to InitSessionHistory() like this: 
  // If we are an in-process browser, we want to set up our session history.
  if (mIsTopLevelContent && mOwnerContent->IsXULElement(nsGkAtoms::browser) &&
      !mOwnerContent->HasAttr(kNameSpaceID_None, nsGkAtoms::disablehistory)) {
    // XXX(nika): Set this up more explicitly?
    mPendingBrowsingContext->InitSessionHistory();
  }
I wonder what's going on around about this code then. Checking with the debugger the answer turns out to be that apparently, nsFrameLoader::GetExtantBrowsingContext() simply doesn't get called either.

From here, things pan out. The EnsureRemoteBrowser() method is called by all of these methods:
  1. nsFrameLoader::GetBrowsingContext()
  2. nsFrameLoader::ShowRemoteFrame()
  3. nsFrameLoader::ReallyStartLoadingInternal()
None of these are static methods and when I place a breakpoint on the nsFrameLoader constructor it doesn't get hit. So it's not possible for any of these methods to be called and there's no point trying to dig any deeper via them.

However this isn't true in ESR 78 where the constructor does get called. It's almost certainly worthwhile finding out about this difference. But unfortunately I'm out of time for today.

I have to wrap things up. As I mentioned previously, I'm taking a break for two weeks to give myself a bit more breathing space as I prepare for FOSDEM, which I'm really looking forward to. If you're travelling to Brussels yourself then I hope to see you there. You'll be able to find me mostly on the Linux on Mobile stand.

When I get back I'll be back to posting these dev diaries again. And as a note to myself, when I do, my first action must be to figure out why nsFrameLoader is being created in ESR 78 but not ESR 91. That might hold the key to why the sessionHistory isn't getting called in ESR 91. And as a last resort, I can always revert commit 140a4164598e0c9ed53.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
24 Jan 2024 : Day 148 #
After much digging around in the code and gecko project structure I eventually decided that the best thing to do is implement a Sailfish-specific version of the SessionStore.jsm module.

Unfortunately this isn't just a case of copying the file over to embedlite-components, because it has some dependencies. These are listed at the top of the file. Let's go through and figure out which ones are already available, which we can remove, which we can copy over directly to embedlite-components and which we have to reimplement ourselves.

Here's the code that relates to the dependencies: 
const { PrivateBrowsingUtils } = ChromeUtils.import(
  "resource://gre/modules/PrivateBrowsingUtils.jsm"
);
const { Services } = ChromeUtils.import("resource://gre/modules/Services.jsm");
const { TelemetryTimestamps } = ChromeUtils.import(
  "resource://gre/modules/TelemetryTimestamps.jsm"
);
const { XPCOMUtils } = ChromeUtils.import(
  "resource://gre/modules/XPCOMUtils.jsm"
);

ChromeUtils.defineModuleGetter(
  this,
  "SessionHistory",
  "resource://gre/modules/sessionstore/SessionHistory.jsm"
);

XPCOMUtils.defineLazyServiceGetters(this, {
  gScreenManager: ["@mozilla.org/gfx/screenmanager;1", "nsIScreenManager"],
});

XPCOMUtils.defineLazyModuleGetters(this, {
  AppConstants: "resource://gre/modules/AppConstants.jsm",
  AsyncShutdown: "resource://gre/modules/AsyncShutdown.jsm",
  BrowserWindowTracker: "resource:///modules/BrowserWindowTracker.jsm",
  DevToolsShim: "chrome://devtools-startup/content/DevToolsShim.jsm",
  E10SUtils: "resource://gre/modules/E10SUtils.jsm",
  GlobalState: "resource:///modules/sessionstore/GlobalState.jsm",
  HomePage: "resource:///modules/HomePage.jsm",
  PrivacyFilter: "resource://gre/modules/sessionstore/PrivacyFilter.jsm",
  PromiseUtils: "resource://gre/modules/PromiseUtils.jsm",
  RunState: "resource:///modules/sessionstore/RunState.jsm",
  SessionCookies: "resource:///modules/sessionstore/SessionCookies.jsm",
  SessionFile: "resource:///modules/sessionstore/SessionFile.jsm",
  SessionSaver: "resource:///modules/sessionstore/SessionSaver.jsm",
  SessionStartup: "resource:///modules/sessionstore/SessionStartup.jsm",
  TabAttributes: "resource:///modules/sessionstore/TabAttributes.jsm",
  TabCrashHandler: "resource:///modules/ContentCrashHandlers.jsm",
  TabState: "resource:///modules/sessionstore/TabState.jsm",
  TabStateCache: "resource:///modules/sessionstore/TabStateCache.jsm",
  TabStateFlusher: "resource:///modules/sessionstore/TabStateFlusher.jsm",
  setTimeout: "resource://gre/modules/Timer.jsm",
});
Heres's a table to summarise. I've ordered them by their current status to help highlight what needs work and what type of work it is.
 
Module Variable Status
gre/modules/PrivateBrowsingUtils.jsm" PrivateBrowsingUtils Available
gre/modules/Services.jsm Services Available
gre/modules/TelemetryTimestamps.jsm TelemetryTimestamps Available
gre/modules/XPCOMUtils.jsm XPCOMUtils Available
gre/modules/sessionstore/SessionHistory.jsm SessionHistory Available
gre/modules/AppConstants.jsm AppConstants Available
gre/modules/AsyncShutdown.jsm AsyncShutdown Available
gre/modules/E10SUtils.jsm E10SUtils Available
gre/modules/sessionstore/PrivacyFilter.jsm PrivacyFilter Available
gre/modules/PromiseUtils.jsm PromiseUtils Available
gre/modules/Timer.jsm setTimeout Available
@mozilla.org/gfx/screenmanager;1 gScreenManager Drop
modules/ContentCrashHandlers.jsm TabCrashHandler Drop
devtools-startup/content/DevToolsShim.jsm DevToolsShim Drop
modules/sessionstore/TabAttributes.jsm TabAttributes Copy
modules/sessionstore/GlobalState.jsm GlobalState Copy
modules/sessionstore/RunState.jsm RunState Copy
modules/BrowserWindowTracker.jsm BrowserWindowTracker Drop
modules/sessionstore/SessionCookies.jsm SessionCookies Drop?
modules/HomePage.jsm HomePage Drop?
modules/sessionstore/SessionFile.jsm SessionFile Copy/drop?
modules/sessionstore/SessionSaver.jsm SessionSaver Copy/drop?
modules/sessionstore/SessionStartup.jsm SessionStartup Copy/drop?
modules/sessionstore/TabState.jsm TabState Copy/drop?
modules/sessionstore/TabStateCache.jsm TabStateCache Copy/drop?
modules/sessionstore/TabStateFlusher.jsm TabStateFlusher Copy/drop?

In addition to the above there's also the SessionStore.jsm file itself.

As you can see there's still a fair bit of uncertainty in the table. But also, quite a large number of the dependencies are already available.

From the code it looks like the functionality is around saving and restoring sessions, including tab data, cookies, window positions and the like. Some of this isn't relevant on Sailfish OS (there's no point saving and restoring window sizes) or is already handled by other parts of the system (cookie storage). In fact, it's not clear that this module is providing any additional functionality that sailfish-browser actually needs.

Given this my focus will be on creating a minimal implementation that doesn't error when called but performs very little functionality in practice. That will hopefully make the task tractable.

It's early in the morning here still, but time for me to start work; so I'll pick this up again tonight.

[...]

It's now late evening and I have just a bit of time to move some files around. I've started by copying the SessionStore.jsm file into the embedlite-components project, alongside the other files I think I can copy without making changes. Apart from SessionStore.jsm, I've tried to copy over only files that don't require dependencies, or where the dependencies are all available. 
$ find . -iname "SessionStore.jsm"
./gecko-dev/browser/components/sessionstore/SessionStore.jsm
$ cp ./gecko-dev/browser/components/sessionstore/SessionStore.jsm \
    ../../embedlite-components/jscomps/
$ cp ./gecko-dev/browser/components/sessionstore/TabAttributes.jsm \
    ../../embedlite-components/jscomps/
$ cp ./gecko-dev/browser/components/sessionstore/GlobalState.jsm \
     ../../embedlite-components/jscomps/
$ cp ./gecko-dev/browser/components/sessionstore/RunState.jsm \
    ../../embedlite-components/jscomps/
I've also been through and removed all of the code that used any of the dropped dependency. And in fact I've gone ahead and dropped all of the modules marked as "Drop" or "Copy/drop" in the table above. Despite the quantity of code in the original files, it really doesn't look like there's much functionality that's needed for sailfish-browser in these scripts. But having the functions available may turn out to be useful at some point in the future and in the meantime if the module just provides methods that don't do anything, then they will at least be successful in suppressing the errors.

The final step is to hook them up into the system so that they get included and can be accessed by other parts of the code. And this is where I hit a problem. The embedlite-components package contains two types of JavaScript entity. The first are in the jscomps folder. These all seem to be components that have a defined interface (they satisfy a "contract") as specified in the EmbedLiteJSComponents.manifest file. Here's an example of the entry for the AboutRedirector component: 
# AboutRedirector.js
component {59f3da9a-6c88-11e2-b875-33d1bd379849} AboutRedirector.js
contract @mozilla.org/network/protocol/about;1?what= {59f3da9a-6c88-11e2-b875-33d1bd379849}
contract @mozilla.org/network/protocol/about;1?what=embedlite {59f3da9a-6c88-11e2-b875-33d1bd379849}
contract @mozilla.org/network/protocol/about;1?what=certerror {59f3da9a-6c88-11e2-b875-33d1bd379849}
contract @mozilla.org/network/protocol/about;1?what=home {59f3da9a-6c88-11e2-b875-33d1bd379849}
Our SessionStore.jsm files can't be added like this because they're not components with defined interfaces in this way. The other type are in the jsscipts folder. These aren't components and the files would fit perfectly in there. But they are all accessed using a particular path and the chrome:// scheme, like this: 
const { NetErrorHelper } = ChromeUtils.import("chrome://embedlite/content/NetErrorHelper.jsm")
This won't work for SessionStore.jsm, which is expected to be accessed like this: 
XPCOMUtils.defineLazyModuleGetters(this, {
  SessionStore: "resource:///modules/sessionstore/SessionStore.jsm",
});
Different location; different approach.

So I'm going to need to find some other way to do this. As it's late, it will take me a while to come up with an alternative. But my immediate thought is that maybe I can just add the missing files in to the EmbedLite build process. It looks like this is being controlled by the embedlite/moz.build file. So I've added the component folder /browser/components/sessionstore into the list of directories there: 
LOCAL_INCLUDES += [
    '!/build',
    '/browser/components/sessionstore',
    '/dom/base',
    '/dom/ipc',
    '/gfx/layers',
    '/gfx/layers/apz/util',
    '/hal',
    '/js/xpconnect/src',
    '/netwerk/base/',
    '/toolkit/components/resistfingerprinting',
    '/toolkit/xre',
    '/widget',
    '/xpcom/base',
    '/xpcom/build',
    '/xpcom/threads',
    'embedhelpers',
    'embedprocess',
    'embedshared',
    'embedthread',
    'modules',
    'utils',
]
I've added the directory, cleaned out the build directory and started off a fresh build to run overnight. Let's see whether that worked in the morning.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
23 Jan 2024 : Day 147 #
Last night I woke up in a mild panic. This happens sometimes, usually when I have a lot going on and I feel like I'm in danger of dropping the ball.

This seems to be my mind's (or maybe my body's?) way of telling me that I need to get my priorities straight. That there's something that I need to get done, resolved or somehow dealt with, and I need to do it urgently or I'll continue to have sleepless nights until I do.

The reason for this particular panic was my FOSDEM preparations, combined with a build up of projects at work that are coming to a head. For FOSDEM I have two talks to prepare (one about this blog, the other related to my work), the Linux on Mobile stand to help organise, the Sailfish community dinner on the Saturday to help organise, support for the HPC, Big Data & Data Science devroom on the Saturday, and also with the Python devroom on the Sunday. It's going to be a crazy busy event. But it's actually the run-up to it and the fact I've to still write my presentations, that's losing me sleep.

It's all fine: it's under control. But in order to prevent it spiralling out of control, I'm going to be taking a break from gecko development for a couple of weeks until things have calmed down. This will slow down development, which of course saddens me because more than anything else I just want ESR 91 to end up in a good, releasable, state. But as others wiser than I am have already cautioned, this also means keeping a positive and healthy state of mind.

I'll finish my last post on Day 149 (that's this Thursday). I'll start back up again on Thursday the 8th February, assuming all goes to plan!

But for the next couple of days there's still development to be done, so let's get straight back to it.

Today I'm still attempting to fix the NS_ERROR_FILE_NOT_FOUND error coming from SessionStore.jsm which I believe may be causing the Back and Forwards buttons in the browser to fail. It's become clear that the SessionStore.jsm file itself is missing (along with a bunch of other files listed in gecko-dev/browser/components/sessionstore/moz.build) but what's not clear is whether the problem is that the files should be there, or that the calls to the methods in this file shouldn't be there.

Overnight while lying in bed I came up with some kind of plan to help move things forwards. The call to the failing method is happening in updateSessionStoreForWindow() and this is only called by the exported method UpdateSessionStoreForWindow(). As far as I can tell this isn't executed by any JavaScript code, but because it has an IPDL interface it's possible for it to be called by C++ code as well.

And sure enough, it's being called twice in WindowGlobalParent.cpp. Once at the end of the WriteFormDataAndScrollToSessionStore() method on line 1260, like this: 
nsresult WindowGlobalParent::WriteFormDataAndScrollToSessionStore(
    const Maybe<FormData>& aFormData, const Maybe<nsPoint>& aScrollPosition,
    uint32_t aEpoch) {
[...]
  return funcs->UpdateSessionStoreForWindow(GetRootOwnerElement(), context, key,
                                            aEpoch, update);
}
And another time at the end of the ResetSessionStore() method on line 1310, like this: 
nsresult WindowGlobalParent::ResetSessionStore(uint32_t aEpoch) {
[...]
  return funcs->UpdateSessionStoreForWindow(GetRootOwnerElement(), context, key,
                                            aEpoch, update);
}
I've placed a breakpoint on these two locations to find out whether these are actually where it's being fired from. If you're being super-observant you'll notice I've not actually placed the breakpoints where I actually want them; I've had to place them earlier in the code (but crucially, still within the same methods). That's because it's not always possible to place breakpoints on the exact line you want.

The debugger will place it on the first line it can after the point you request. Because both of the cases I'm interested in are right at the end of the methods they're called in, when I attempt to put a breakpoint on the exact line the debugger places it instead in the next method along in the source code. That isn't much use for what I needed.

Hence I've placed them a little earlier in the code instead: on the first lines where they actually stick. 
bash-5.0$ EMBED_CONSOLE=1 MOZ_LOG="EmbedLite:5" gdb sailfish-browser
(gdb) b WindowGlobalParent.cpp:1260
Breakpoint 5 at 0x7fbbc8e31c: file dom/ipc/WindowGlobalParent.cpp, line 1260.
(gdb) b WindowGlobalParent.cpp:1294
Breakpoint 9 at 0x7fbbc8e688: file dom/ipc/WindowGlobalParent.cpp, line 1294.
(gdb) r
[...]
JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 541:
    NS_ERROR_FILE_NOT_FOUND: 
CONSOLE message:
[JavaScript Error: "NS_ERROR_FILE_NOT_FOUND: " {file:
    "resource:///modules/sessionstore/SessionStore.jsm" line: 541}]
@resource:///modules/sessionstore/SessionStore.jsm:541:3
SSF_updateSessionStoreForWindow@resource://gre/modules/
    SessionStoreFunctions.jsm:120:5
UpdateSessionStoreForStorage@resource://gre/modules/
    SessionStoreFunctions.jsm:54:35

Thread 10 "GeckoWorkerThre" hit Breakpoint 5, mozilla::dom::WindowGlobalParent::
    WriteFormDataAndScrollToSessionStore (this=this@entry=0x7f81164520, 
    aFormData=..., aScrollPosition=..., aEpoch=0)
    at dom/ipc/WindowGlobalParent.cpp:1260
1260      windowState.mHasChildren.Construct() = !context->Children().IsEmpty();
(gdb) n
709     ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/Span.h:
        No such file or directory.
(gdb) 
1260      windowState.mHasChildren.Construct() = !context->Children().IsEmpty();
(gdb) 
1262      JS::RootedValue update(jsapi.cx());
(gdb) 
1263      if (!ToJSValue(jsapi.cx(), windowState, &update)) {
(gdb) 
1267      JS::RootedValue key(jsapi.cx(), context->Top()->PermanentKey());
(gdb) 
1269      return funcs->UpdateSessionStoreForWindow(GetRootOwnerElement(),
          context, key,
(gdb) n
1297    ${PROJECT}/obj-build-mer-qt-xr/dist/include/js/RootingAPI.h:
        No such file or directory.
(gdb) 
JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 541:
    NS_ERROR_FILE_NOT_FOUND: 
1267      JS::RootedValue key(jsapi.cx(), context->Top()->PermanentKey());
(gdb) c
Continuing.
It's clear from the above that the WriteFormDataAndScrollToSessionStore() method is being called and is then going on to call the missing JavaScript method. We can even see the error coming from the JavaScript as we step out of the calling method.

You'll notice from the output that there's another — earlier — NS_ERROR_FILE_NOT_FOUND error before the breakpoint hits. This is coming from a different spot: line 120 of SessionStoreFunctions.jsm rather than the line 105 we were looking at here.

This new error is called from line 54 of the same file (we can see from from the backtrace in the log output) which is in the UpdateSessionStoreForStorage() method in the file. So where is this being called from? 
$ grep -rIn "UpdateSessionStoreForStorage" * --include="*.js" \
    --include="*.jsm" --include="*.cpp" --exclude-dir="obj-build-mer-qt-xr"
gecko-dev/docshell/base/CanonicalBrowsingContext.cpp:2233:
    return funcs->UpdateSessionStoreForStorage(Top()->GetEmbedderElement(), this,
gecko-dev/docshell/base/CanonicalBrowsingContext.cpp:2255:
    void CanonicalBrowsingContext::UpdateSessionStoreForStorage(
gecko-dev/dom/storage/SessionStorageManager.cpp:854:
    CanonicalBrowsingContext::UpdateSessionStoreForStorage(
gecko-dev/toolkit/components/sessionstore/SessionStoreFunctions.jsm:47:
    function UpdateSessionStoreForStorage(
gecko-dev/toolkit/components/sessionstore/SessionStoreFunctions.jsm:66:
    "UpdateSessionStoreForStorage",
From this we can see it's being called in a few places, all of them from C++ code. Again, that means we can explore them with gdb using breakpoints. Let's give this a go as well. 
(gdb) break CanonicalBrowsingContext.cpp:2213
Breakpoint 7 at 0x7fbc7c6abc: file docshell/base/CanonicalBrowsingContext.cpp,
    line 2213.
(gdb) r
[...]

Thread 10 "GeckoWorkerThre" hit Breakpoint 7, mozilla::dom::
    CanonicalBrowsingContext::WriteSessionStorageToSessionStore
    (this=0x7f80b6dee0, aSesssionStorage=..., aEpoch=0)
    at docshell/base/CanonicalBrowsingContext.cpp:2213
2213      AutoJSAPI jsapi;
(gdb) bt
#0  mozilla::dom::CanonicalBrowsingContext::WriteSessionStorageToSessionStore
    (this=0x7f80b6dee0, aSesssionStorage=..., aEpoch=0)
    at docshell/base/CanonicalBrowsingContext.cpp:2213
#1  0x0000007fbc7c6f54 in mozilla::dom::CanonicalBrowsingContext::
    <lambda(const mozilla::MozPromise<nsTArray<mozilla::dom::SSCacheCopy>,
    mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&)>::
    operator() (valueList=..., __closure=0x7f80bd70c8)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/Variant.h:768
#2  mozilla::MozPromise<nsTArray<mozilla::dom::SSCacheCopy>, mozilla::ipc::
    ResponseRejectReason, true>::InvokeMethod<mozilla::dom::
    CanonicalBrowsingContext::UpdateSessionStoreSessionStorage(const
    std::function<void()>&)::<lambda(const mozilla::MozPromise<nsTArray
    <mozilla::dom::SSCacheCopy>, mozilla::ipc::ResponseRejectReason, true>::
    ResolveOrRejectValue&)>, void (mozilla::dom::CanonicalBrowsingContext::
    UpdateSessionStoreSessionStorage(const std::function<void()>&)::
    <lambda(const mozilla::MozPromise<nsTArray<mozilla::dom::SSCacheCopy>,
    mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&)>::*)
    (const mozilla::MozPromise<nsTArray<mozilla::dom::SSCacheCopy>,
    mozilla::ipc::ResponseRejectReason, true>::ResolveOrRejectValue&) const,
    mozilla::MozPromise<nsTArray<mozilla::dom::SSCacheCopy>, mozilla::ipc::
    ResponseRejectReason, true>::ResolveOrRejectValue>
    (aValue=..., aMethod=<optimized out>, 
[...]
#25 0x0000007fb78b289c in ?? () from /lib64/libc.so.6
(gdb) n
[LWP 8594 exited]
2214      if (!jsapi.Init(wrapped->GetJSObjectGlobal())) {
(gdb) 
2218      JS::RootedValue key(jsapi.cx(), Top()->PermanentKey());
(gdb) 
2220      Record<nsCString, Record<nsString, nsString>> storage;
(gdb) 
2221      JS::RootedValue update(jsapi.cx());
(gdb) 
2223      if (!aSesssionStorage.IsEmpty()) {
(gdb) 
2230        update.setNull();
(gdb) 
2233      return funcs->UpdateSessionStoreForStorage(Top()->
          GetEmbedderElement(), this,
(gdb) 
1297    ${PROJECT}/obj-build-mer-qt-xr/dist/include/js/RootingAPI.h:
        No such file or directory.
(gdb) 
JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 541:
    NS_ERROR_FILE_NOT_FOUND: 
2221      JS::RootedValue update(jsapi.cx());
(gdb) 
2220      Record<nsCString, Record<nsString, nsString>> storage;
(gdb)
This new breakpoint is hit and once again, stepping through the code shows the problem method being called and also triggering the JavaScript error we're concerned about.

This is all good stuff. Looking through the ESR 78 code there doesn't appear to be anything equivalent in CanonicalBrowsingContext.cpp. But now that I know this is where the problem is happening, I can at least find the commit that introduced the changes and use that to find out more. Here's the output from git blame, but please forgive the terrible formatting: it's very hard to line-wrap this output cleanly. 
$ git blame docshell/base/CanonicalBrowsingContext.cpp \
    -L :WriteSessionStorageToSessionStore
dd51467 (Andreas Farre 2021-05-26 2204) nsresult CanonicalBrowsingContext::
                                        WriteSessionStorageToSessionStore(
dd51467 (Andreas Farre 2021-05-26 2205)     const nsTArray<SSCacheCopy>&
                                            aSesssionStorage, uint32_t aEpoch) {
dd51467 (Andreas Farre 2021-05-26 2206)   nsCOMPtr<nsISessionStoreFunctions> funcs =
dd51467 (Andreas Farre 2021-05-26 2207)       do_ImportModule("resource://gre/
                                              modules/SessionStoreFunctions.jsm");
dd51467 (Andreas Farre 2021-05-26 2208)   if (!funcs) {
dd51467 (Andreas Farre 2021-05-26 2209)     return NS_ERROR_FAILURE;
dd51467 (Andreas Farre 2021-05-26 2210)   }
dd51467 (Andreas Farre 2021-05-26 2211) 
dd51467 (Andreas Farre 2021-05-26 2212)   nsCOMPtr<nsIXPConnectWrappedJS>
                                          wrapped = do_QueryInterface(funcs);
dd51467 (Andreas Farre 2021-05-26 2213)   AutoJSAPI jsapi;
dd51467 (Andreas Farre 2021-05-26 2214)   if (!jsapi.Init(wrapped->
                                            GetJSObjectGlobal())) {
dd51467 (Andreas Farre 2021-05-26 2215)     return NS_ERROR_FAILURE;
dd51467 (Andreas Farre 2021-05-26 2216)   }
dd51467 (Andreas Farre 2021-05-26 2217) 
2b70b9d (Kashav Madan  2021-06-26 2218)   JS::RootedValue key(jsapi.cx(),
                                            Top()->PermanentKey());
2b70b9d (Kashav Madan  2021-06-26 2219) 
dd51467 (Andreas Farre 2021-05-26 2220)   Record<nsCString, Record<nsString,
                                            nsString>> storage;
dd51467 (Andreas Farre 2021-05-26 2221)   JS::RootedValue update(jsapi.cx());
dd51467 (Andreas Farre 2021-05-26 2222) 
dd51467 (Andreas Farre 2021-05-26 2223)   if (!aSesssionStorage.IsEmpty()) {
dd51467 (Andreas Farre 2021-05-26 2224)     SessionStoreUtils::
                                              ConstructSessionStorageValues(this,
                                              aSesssionStorage,
dd51467 (Andreas Farre 2021-05-26 2225)                                storage);
dd51467 (Andreas Farre 2021-05-26 2226)     if (!ToJSValue(jsapi.cx(), storage,
                                              &update)) {
dd51467 (Andreas Farre 2021-05-26 2227)       return NS_ERROR_FAILURE;
dd51467 (Andreas Farre 2021-05-26 2228)     }
dd51467 (Andreas Farre 2021-05-26 2229)   } else {
dd51467 (Andreas Farre 2021-05-26 2230)     update.setNull();
dd51467 (Andreas Farre 2021-05-26 2231)   }
dd51467 (Andreas Farre 2021-05-26 2232) 
2b70b9d (Kashav Madan  2021-06-26 2233)   return funcs->
                                            UpdateSessionStoreForStorage(
                                            Top()->GetEmbedderElement(), this,
2b70b9d (Kashav Madan  2021-06-26 2234)                    key, aEpoch, update);
dd51467 (Andreas Farre 2021-05-26 2235) }
dd51467 (Andreas Farre 2021-05-26 2236)
If you can get past the terrible formatting you should be able to see there are two commits of interest here. The first is dd51467c228cb from Andreas Farre and the second which was layered on top is 2b70b9d821c8e from Kashav Madan. Let's find out more about them both. 
$ git log -1 dd51467c228cb
commit dd51467c228cb5c9ec9d9efbb6e0339037ec7fd5
Author: Andreas Farre <farre@mozilla.com>
Date:   Wed May 26 07:14:06 2021 +0000

    Part 7: Bug 1700623 - Make session storage session store work with Fission.
        r=nika
    
    Use the newly added session storage data getter to access the session
    storage in the parent and store it in session store without a round
    trip to content processes.
    
    Depends on D111433
    
    Differential Revision: https://phabricator.services.mozilla.com/D111434

For some reason the Phabricator link doesn't work for me, but we can still see the revision directly in the repository.
$ git log -1 2b70b9d821c8e commit 2b70b9d821c8eaf0ecae987cfc57e354f0f9cc20 Author: Kashav Madan <kshvmdn@gmail.com> Date: Sat Jun 26 20:25:29 2021 +0000 Bug 1703692 - Store the latest embedder's permanent key on CanonicalBrowsingContext, r=nika,mccr8 And include it in Session Store flushes to avoid dropping updates in case the browser is unavailable. Differential Revision: https://phabricator.services.mozilla.com/D118385 There aren't many clues in these changes, in particular there's no hint of how the build system was changed to have these files included. However, digging around in this code has given me a better understanding of the structure and purpose of the different directories.

It seems to me that, in essence, the gecko-dev/browser/components/ directory where the missing files can be found contains modules that relate to the browser chrome and Firefox user interface, rather than the rendering engine. Typically this kind of content would be replicated on Sailfish OS by adding amended versions into the embedlite-components project. If it's browser-specific material, that would make sense.

As an example, in Firefox we can find AboutRedirector.h and AboutRedirector.cpp files , whereas on Sailfish OS there's a replacement AboutRedirectory.js file in embedlite-components. Similarly Firefox has a DownloadsManager.jsm file that can be found in gecko-dev/browser/components/newtab/lib/. This seems to be replaced by EmbedliteDownloadManager.js in embedlite-components. Both have similar functionality based on the names of the methods contained in them, but the implementations are quite different.

Assuming this is correct, probably the right way to tackle the missing SessionStore.jsm and related files would be to move copies into embedlite-components. They'll need potentially quite big changes to align them with sailfish-browser, although hopefully this will largely be removing functionality that has already been implemented elsewhere (for example cookie save and restore).

I think I'll give these changes a go tomorrow.

Another thing I've pretty-much concluded while looking through this code is that it looks like it probably has nothing to do with the issue that's blocking the Back and Forward buttons from working after all. So I'll also need to make a separate task to track down the real source of that problem.

Right now I have a stack of tasks: SessionStore; Back/Forward failures; DuckDuckGo rendering. I mustn't lose site of the fact that the real goal right now is to get DuckDuckGo rendering correctly. The other tasks are secondary, albeit with DuckDuckGo rendering potentially dependent on them.

That's it for today. More tomorrow and Thursday, but then a bit of a break.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
22 Jan 2024 : Day 146 #
I'm still in the process of fixing the Sec-Fetch-* headers. The data I collected yesterday resulted in a few conclusions:
  1. Opening a URL at the command line with a tab that was already open gives odd results.
  2. Opening a URL as a homepage gives odd results.
  3. The Back and Forwards buttons are broken so couldn't be tested.
  4. I didn't get time to test the JavaScript case.
I'm going to tackle the JavaScript case first today. It turns out even for this one situation there are at least two cases to consider:
  1. Open a URL using JavaScript simulating an HREF selection.
  2. Open a URL using a JavaScript redirect.
To test this I've created a minimal web page with a couple of links that perform these actions. Here's the content of the page: 
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <meta name="viewport" content="width=device-width, user-scalable=no"/>
    <script type="text/javascript">
      function reloadHref(url) {
        setTimeout(() => window.location.href = url, 2000);
      }
      function reloadRedirect(url) {
        setTimeout(() => window.location.replace(url), 2000);
      }
    </script>
  </head>
  <body>
    <p><a href="javascript:reloadHref('https://duckduckgo.com');">
      Simulate an HREF selection
    </a></p>
    <p><a href="javascript:reloadRedirect('https://duckduckgo.com');">
      Simulate a redirect
    </a></p>
  </body>
</html>
Pretty straightforward stuff, but should do the trick. This allows me to test the effects of the URL being changed and record the results. In fact, let's get straight to the results. Here's what I found out by testing using this page:
 
Situation Expected Flag set
Open a URL using JavaScript simulating a HREF selection. 0 0
Open a URL using a JavaScript redirect. 0 0

I do note however also notice that DuckDuckGo doesn't load correctly for these cases, so presumably there's still a problem with the headers here. I'll have to come back to that.

The inability to use the Back or Forwards buttons is also beginning to cause me trouble in day-to-day use, so now might be the time to fix that as well. Once I have I can test the remaining cases.

My suspicion is that the reason they don't work is related to this error that appears periodically when using the browser: 
JavaScript error: resource://gre/modules/SessionStoreFunctions.jsm, line 105: NS_ERROR_FILE_NOT_FOUND:
Here's the code in the file that's generating this error: 
    SessionStore.updateSessionStoreFromTablistener(
      aBrowser,
      aBrowsingContext,
      aPermanentKey,
      { data: { windowstatechange: aData }, epoch: aEpoch }
    );
It could be an error happening inside SessionStore.updateSessionStoreFromTablistener() but two reasons make me think this is unlikely. First the error message clearly targets the calling location and if the error were inside this method I'd expect the error message to reflect that instead. Second there isn't anything obvious in the updateSessionStoreFromTablistener() body that might be causing an error like this. No obvious file accesses or anything like that.

A different possibility is that this, at the top of the SessionStoreFunctions.jsm file, is causing the problem: 
XPCOMUtils.defineLazyModuleGetters(this, {
  SessionStore: "resource:///modules/sessionstore/SessionStore.jsm",
});
This is a lazy getter, meaning that an attempt will be made to load the resource only at the point where a method from the module is used. Could it be that the SessionStore.jsm file is inaccessible? Then when a method from it is called the JavaScript interpreter tries to load the code in and fails, triggering the error.

A quick search inside the omni archive suggests this file is indeed missing: 
$ find . -iname "SessionStore.jsm"
$ find . -iname "SessionStoreFunctions.jsm"
./omni/modules/SessionStoreFunctions.jsm
$ find . -iname "Services.jsm"
./omni/modules/Services.jsm
As we can see, in contrast the SessionStoreFunctions.jsm and Services.jsm files are both present and correct. Well, present at least. To test out the theory that this is the problem I've parachuted the file into omni. First from my laptop: 
$ scp gecko-dev/browser/components/sessionstore/SessionStore.jsm \
    defaultuser@10.0.0.116:./omni/modules/sessionstore/
SessionStore.jsm                              100%  209KB   1.3MB/s   00:00    
[...]
And then on my phone: 
$ ./omni.sh pack
Omni action: pack
Packing from: ./omni
Packing to:   /usr/lib64/xulrunner-qt5-91.9.1
This hasn't fixed the Back and Forwards buttons, but it has resulted in a new error. The fact that this is error is now coming from inside SessionStore.jsm is encouraging. 
JavaScript error: chrome://embedlite/content/embedhelper.js, line 259:
    TypeError: sessionHistory is null
JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 541:
    NS_ERROR_FILE_NOT_FOUND:
Line 541 of SessionStore.jsm looks like this: 
  _globalState: new GlobalState(),
This also looks lazy-getter-related, since the only other reference to GlobalState() in this file is at the top, in this chunk of lazy-getter code: 
XPCOMUtils.defineLazyModuleGetters(this, {
  AppConstants: "resource://gre/modules/AppConstants.jsm",
  AsyncShutdown: "resource://gre/modules/AsyncShutdown.jsm",
  BrowserWindowTracker: "resource:///modules/BrowserWindowTracker.jsm",
  DevToolsShim: "chrome://devtools-startup/content/DevToolsShim.jsm",
  E10SUtils: "resource://gre/modules/E10SUtils.jsm",
  GlobalState: "resource:///modules/sessionstore/GlobalState.jsm",
  HomePage: "resource:///modules/HomePage.jsm",
  PrivacyFilter: "resource://gre/modules/sessionstore/PrivacyFilter.jsm",
  PromiseUtils: "resource://gre/modules/PromiseUtils.jsm",
  RunState: "resource:///modules/sessionstore/RunState.jsm",
  SessionCookies: "resource:///modules/sessionstore/SessionCookies.jsm",
  SessionFile: "resource:///modules/sessionstore/SessionFile.jsm",
  SessionSaver: "resource:///modules/sessionstore/SessionSaver.jsm",
  SessionStartup: "resource:///modules/sessionstore/SessionStartup.jsm",
  TabAttributes: "resource:///modules/sessionstore/TabAttributes.jsm",
  TabCrashHandler: "resource:///modules/ContentCrashHandlers.jsm",
  TabState: "resource:///modules/sessionstore/TabState.jsm",
  TabStateCache: "resource:///modules/sessionstore/TabStateCache.jsm",
  TabStateFlusher: "resource:///modules/sessionstore/TabStateFlusher.jsm",
  setTimeout: "resource://gre/modules/Timer.jsm",
});
Sure enough, when I check, the GlobalState.jsm file is missing. It looks like these missing files are ones referenced in gecko-dev/browser/components/sessionstore/moz.build: 
EXTRA_JS_MODULES.sessionstore = [
    "ContentRestore.jsm",
    "ContentSessionStore.jsm",
    "GlobalState.jsm",
    "RecentlyClosedTabsAndWindowsMenuUtils.jsm",
    "RunState.jsm",
    "SessionCookies.jsm",
    "SessionFile.jsm",
    "SessionMigration.jsm",
    "SessionSaver.jsm",
    "SessionStartup.jsm",
    "SessionStore.jsm",
    "SessionWorker.js",
    "SessionWorker.jsm",
    "StartupPerformance.jsm",
    "TabAttributes.jsm",
    "TabState.jsm",
    "TabStateCache.jsm",
    "TabStateFlusher.jsm",
]
It's not at all clear to me why these files aren't being included. The problem must be arising because either they're not being included when they should be, or they're being accessed when they shouldn't be.

But it's late now, so I'm going to have to figure that out tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
21 Jan 2024 : Day 145 #
Yesterday was a light day of gecko development (heavy on everything else but light on gecko). I managed to update the user agent overrides but not a lot else.

The one thing I did do was think about next steps, which brings us to today. To recap, the DuckDuckGo main page is now working. The search page inexplicably has no search results on it and so needs fixing. But the first thing I need to do is check whether the user interaction flags are propagating properly. My working assumption is that in the cases where they're needed they're being set. What I'm less certain about is whether they're not being set when they're not needed.

The purpose of the Sec-Fetch-* headers is to allow the browser to work in collaboration with the server. The user doesn't necessarily trust the page they're viewing and the server doesn't necessarily trust the browser. But the user should trust the browser. And the user should trust the browser to send the correct Sec-Fetch-* headers to the server. Assuming they're set correctly a trustworthy site can then act on them accordingly; for example, by only showing private data when the page isn't being displayed in an iframe, say.

Anyway, the point is, setting the value of these headers is a security feature. The implicit contract between user and browser requires that they're set correctly and the user trusts the browser will do this. The result of not doing so could make it easier for attackers to trick the user. So getting the flags set correctly is really important.

When it comes to understanding the header values and the flags that control them, the key gateway is EmbedLiteViewChild::RecvLoadURL(). The logic for deciding whether to set the flags happens before this is called and all of the logic that uses the flag happens after it. So I'll place a breakpoint on this method and check the value of the flag in various situations.

Which situations? Here are the ones I can think of where the flag should be set to true:
  1. Open a URL at the command line with no existing tabs.
  2. Open a URL at the command line with existing tabs.
  3. Open a URL via D-Bus with no existing tabs.
  4. Open a URL via D-Bus with existing tabs.
  5. Open a URL using xdg-open with no existing tags.
  6. Open a URL using xdg-open with existing tags.
And for the following situations the flag should be set to false.
  1. Open a URL as the homepage.
  2. Enter a URL in the address bar.
  3. Open an item from the history.
  4. Open a bookmark.
  5. Select a link on a page.
  6. Open a URL using JavaScript.
  7. Open a page using the Back button.
  8. Open a page using the Forwards button.
  9. Reloading a page.
Here are the results of one debugging cycle. I've skipped the others that are similar to this. 
$ gdb sailfish-browser
(gdb) b EmbedLiteViewChild::RecvLoadURL
Function "EmbedLiteViewChild::RecvLoadURL" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (EmbedLiteViewChild::RecvLoadURL) pending.
(gdb) r https://duckduckgo.com

Thread 8 "GeckoWorkerThre" hit Breakpoint 1, mozilla::embedlite::
    EmbedLiteViewChild::RecvLoadURL (this=0x7f88ad1c60, url=..., 
    aFromExternal=@0x7f9f3d3598: true) at mobile/sailfishos/embedshared/
    EmbedLiteViewChild.cpp:482
482     {
(gdb) p aFromExternal
$2 = (const bool &) @0x7f9f3d3598: true
(gdb) n
483       LOGT("url:%s", NS_ConvertUTF16toUTF8(url).get());
(gdb) n
867     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:
    No such file or directory.
(gdb) n
487       if (Preferences::GetBool("keyword.enabled", true)) {
(gdb) n
493       if (aFromExternal) {
(gdb) n
497       LoadURIOptions loadURIOptions;
(gdb) n
498       loadURIOptions.mTriggeringPrincipal = nsContentUtils::GetSystemPrincipal();
(gdb) p /x flags
$3 = 0x341000
(gdb) p/x flags & nsIWebNavigation::LOAD_FLAGS_FIXUP_SCHEME_TYPOS
$6 = 0x200000
(gdb) p/x flags & nsIWebNavigation::LOAD_FLAGS_ALLOW_THIRD_PARTY_FIXUP
$7 = 0x100000
(gdb) p/x flags & nsIWebNavigation::LOAD_FLAGS_DISALLOW_INHERIT_PRINCIPAL
$8 = 0x40000
(gdb) p/x flags & nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL
$9 = 0x1000
(gdb) p /x flags - (nsIWebNavigation::LOAD_FLAGS_ALLOW_THIRD_PARTY_FIXUP
    | nsIWebNavigation::LOAD_FLAGS_FIXUP_SCHEME_TYPOS
    | nsIWebNavigation::LOAD_FLAGS_DISALLOW_INHERIT_PRINCIPAL
    | nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL)
$10 = 0x0
(gdb)
In a couple of places (selecting links and reloading the page) the EmbedLiteViewChild::RecvLoadURL() method doesn't get called. For those cases I put a breakpoint on LoadInfo::GetLoadTriggeredFromExternal() instead. The process looks a little different: 
(gdb) disable break
(gdb) break GetLoadTriggeredFromExternal
Breakpoint 2 at 0x7fb9d3430c: GetLoadTriggeredFromExternal. (2 locations)
(gdb) c
Continuing.

Thread 8 "GeckoWorkerThre" hit Breakpoint 2, mozilla::net::LoadInfo::
    GetLoadTriggeredFromExternal (this=0x7f89170cf0, 
    aLoadTriggeredFromExternal=0x7f9f3d3150) at netwerk/base/LoadInfo.cpp:1478
1478      *aLoadTriggeredFromExternal = mLoadTriggeredFromExternal;
(gdb) p mLoadTriggeredFromExternal
$1 = false
(gdb)
I've been through and checked the majority of the cases separately. Here's a summary of the results once I apply these processes for all of the cases.
 
Situation Expected Flag set
Open a URL at the command line with no existing tabs. 1 1
Open a URL at the command line with the same tab open. 1 0
Open a URL at the command line with a different tab open. 1 1
Open a URL via D-Bus with no existing tabs. 1 1
Open a URL via D-Bus with the same tab open. 1 No effect
Open a URL via D-Bus with a different tab open. 1 1
Open a URL using xdg-open with no existing tags. 1 1
Open a URL using xdg-open with the same tab open. 1 No effect
Open a URL using xdg-open with a different tab open. 1 1
Open a URL as the homepage. 0 1
Enter a URL in the address bar. 0 0
Open an item from the history. 0 0
Open a bookmark. 0 0
Select a link on a page. 0 0
Open a URL using JavaScript. 0 Not tested
Open a page using the Back button. 0 Unavailable
Open a page using the Forwards button. 0 Unavailable
Reloading a page. 0 0

There are some notable entries in the table although broadly speaking the results are what I was hoping for. For example, when using D-Bus or xdg-open to open the same website that's already available, there is no effect. I hadn't expected this, but having now seen the behaviour in action, it makes perfect sense and looks correct. For the case of opening a URL via the command line with the same tab open, I'll need to look in to whether some other flag should be set instead; but on the face of it, this looks like something that may need fixing.

Similarly for opening a URL as the home page. I think the result is the reverse of what it should be, but I need to look into this more to check.

The forward and back button interactions are marked as "Unavailable". That's because the back and forward functionality are currently broken. I'm hoping that fixing Issue 1024 will also restore this functionality, after which I'll need to test this again.

Finally I didn't get time to test the JavaScript case. I'll have to do that tomorrow.

So a few things still to fix, but hopefully over the next couple of days these can all be ironed out.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
20 Jan 2024 : Day 144 #
It was gratifying to finally see the DuckDuckGo main search page appearing with an ESR 91 build of the browser. Even the search suggestions are working nicely. But there's trouble beneath the surface. Dig just a little further and it turns out the search results page shows no results. In terms of search engine utility, this is sub-optimal. I'll need to look in to this and fix it. But it's not the first task I need to tackle today. Although the front page at least looked like it was working yesterday, it relied on some changes that still need to be fully implemented and checked. The easy task is adding an override to the user agent string override list. What I found yesterday is that while an ESR 91 user agent string only works with the correct Sec-Fetch headers, it also has to be the mobile version of the user agent. This works: 
Mozilla/5.0 (Mobile; rv:91.0) Gecko/91.0 Firefox/91.0
This fails: 
Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Firefox/91.0
All of these things have to align. Updating the ua-update.json.in file in the sailfish-browser repository is straightforward. Just make the change, run the preprocess-ua-update-json batch file and copy the result into the correct folder. 
$ git diff data/ua-update.json.in
diff --git a/data/ua-update.json.in b/data/ua-update.json.in
index 584720c0..338b8faf 100644
--- a/data/ua-update.json.in
+++ b/data/ua-update.json.in
@@ -116,0 +116,1 @@
+  "duckduckgo.com": "Mozilla/5.0 (Mobile; rv:91.0) Gecko/91.0 Firefox/91.0"

$ ./preprocess-ua-update-json
[sailfishos-esr91 fbf5b15e] [user-agent] Update preprocessed user agent overrides
 1 file changed, 2 insertions(+), 1 deletion(-)
$ ls ua/
38.8.0  45.9.1  52.9.1  60.0  60.9.1  78.0  91.0
$ cp ua-update.json ua/91.0/
The second task is to check the code changes I made over the last couple of days. I added flags that pass certain user interaction signals on to the engine, such as whether an action was user-performed or not. Yesterday I checked using both gdb and by observing the resulting Sec-Fetch-* headers that the flags were making their way through the system correctly. However what I didn't check — and what I need to check still — is that the flags are correct when different paths are used to get there. For example, the system needs to distinguish between entering a URL in the toolbar and triggering a URL via D-Bus. The resulting request headers should be the same, but the logic for how we get to the same place is different. This is a debugging task. Unfortunately time has run away today already, so I'll have to pick up on the actual task of debugging tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
19 Jan 2024 : Day 143 #
Over the last couple of days now I've been building versions of gecko and its related packages that support the LOAD_FLAGS_FROM_EXTERNAL flag. According to comments in the code, when this flag is set the engine considers the request to be the result of user-interaction, so I'm hoping this change will help ensure the >Sec-Fetch headers are given the correct values.

But of course not all page loads are triggered externally. There are other flags to identify other types of user interaction which I'll need to fix as well. But one thing at a time. First of all I need to find out if the changes I've made to the code are having any effect.

The changes affect sixteen packages and comprise 771 MiB of data in all (including the debug packages), which I'm now copying over to my device so I can install and test them. Just doing this can take quite a while!

[...]

I've installed them and run them and they don't fix the problem. But as soon as I ran them I remembered that I've not yet hooked up things on the QML side to make them work. I still have a bit more coding to do before we'll see any good results.

[...]

I've updated the QML so that the flag is now passed in for the routes I could find that relate to page loads being triggered externally. There are some potential gotchas here: am I catching all of the situations in which this can happen? Are some of the paths used by other routes as well, such as entering a URL at the address bar? Once I've confirmed that these changes are having an effect I'll need to go back and check these other things as well.

First things first. Let's find out whether the flag is being applied in the case where a URL is passed on the command line. 
$ gdb sailfish-browser
[...]
b DeclarativeTabModel::newTab
Breakpoint 1 at 0x6faf8: DeclarativeTabModel::newTab. (2 locations)
(gdb) r https://www.duckduckgo.com
[...]
Thread 1 "sailfish-browse" hit Breakpoint 1, DeclarativeTabModel::newTab
    (this=0x55559d7b80, url=..., fromExternal=true)
    at ../history/declarativetabmodel.cpp:196
196         return newTab(url, 0, 0, false, fromExternal);
(gdb) p fromExternal
$1 = true
(gdb) b EmbedLiteViewChild::RecvLoadURL
Breakpoint 2 at 0x7fbcb105f0: file mobile/sailfishos/embedshared/
    EmbedLiteViewChild.cpp, line 482.
(gdb) n

Thread 8 "GeckoWorkerThre" hit Breakpoint 2, mozilla::embedlite::
    EmbedLiteViewChild::RecvLoadURL (this=0x7f88690d50, url=..., 
    aFromExternal=@0x7f9f3d3598: true)
    at mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:482
482     {
(gdb) n
483       LOGT("url:%s", NS_ConvertUTF16toUTF8(url).get());
(gdb) n
867     ${PROJECT}/gecko-dev/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h: No such file or directory.
(gdb) n
487       if (Preferences::GetBool("keyword.enabled", true)) {
(gdb) n
493       if (aFromExternal) {
(gdb) n
497       LoadURIOptions loadURIOptions;
(gdb) p aFromExternal
$2 = (const bool &) @0x7f9f3d3598: true
(gdb) n
498       loadURIOptions.mTriggeringPrincipal = nsContentUtils::GetSystemPrincipal();
(gdb) n
499       loadURIOptions.mLoadFlags = flags;
(gdb) p /x flags
$4 = 0x341000
(gdb) p /x nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL
$5 = 0x1000
(gdb) p /x (flags & nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL)
$6 = 0x1000
Great! The flag is being set in the user-interface and is successfully passing through from sailfish-browser to qtmozembed and on to the EmbedLite wrapper for gecko itself. Let's see if it makes it all the way to the request header methods. 
(gdb) b SecFetch::AddSecFetchUser
Breakpoint 3 at 0x7fbba7b680: file dom/security/SecFetch.cpp, line 333.
(gdb) b GetLoadTriggeredFromExternal
Breakpoint 4 at 0x7fb9d3430c: GetLoadTriggeredFromExternal. (5 locations)
(gdb) c
Continuing.
[New LWP 8921]

Thread 8 "GeckoWorkerThre" hit Breakpoint 4, nsILoadInfo::
    GetLoadTriggeredFromExternal (this=0x7f88eabce0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h:664
664     ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsILoadInfo.h: No such file or directory.
(gdb) n

Thread 8 "GeckoWorkerThre" hit Breakpoint 4, mozilla::net::LoadInfo::
    GetLoadTriggeredFromExternal (this=0x7f88eabce0, 
    aLoadTriggeredFromExternal=0x7f9f3d3150) at netwerk/base/LoadInfo.cpp:1478
1478      *aLoadTriggeredFromExternal = mLoadTriggeredFromExternal;
(gdb) p mLoadTriggeredFromExternal
$9 = true
(gdb) c
[...]
mozilla::dom::SecFetch::AddSecFetchUser
    (aHTTPChannel=aHTTPChannel@entry=0x7f88eabed0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/DebugOnly.h:97
97      ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/DebugOnly.h:
        No such file or directory.
(gdb) 
350       nsAutoCString user("?1");
(gdb)
The Sec-Fetch-User flag is now being set correctly. Hooray! It's clear from this that the LOAD_FLAGS_FROM_EXTERNAL flag is making it through all the way to the place where the flag values are set. So let's check what the actual values are using the debug output on the console: 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : none
        Sec-Fetch-User : ?1
        If-None-Match : "65a81e81-4858"
That looks like just the set of Sec-Fetch flags we wanted.

So it's disappointing to discover that the page is still not rendering. Why is could this be?

I set the user agent to the ESR 91 value I used previously: 
  "duckduckgo.com": "Mozilla/5.0 (Mobile; rv:91.0) Gecko/91.0 Firefox/91.0"
Close the browser; clear out the cache; try again: 
$ rm -rf ~/.local/share/org.sailfishos/browser/.mozilla/cache2/ \
    ~/.local/share/org.sailfishos/browser/.mozilla/startupCache/ \
    ~/.local/share/org.sailfishos/browser/.mozilla/cookies.sqlite 
$ sailfish-browser https://duckduckgo.com/
 
Three screenshots: the DuckDuckGo main page showing in ESR 91; search suggestions for the search term 'DuckDuckGo'; the search results page with no search results on it

The last thing I see before I head to bed is the DuckDuckGo logo staring back at me from the screen. There is a glitch with search though. The search action works, in the sense that it takes me to the result page, but no results are shown. So there's still some more work to be done.

Nevertheless I'll sleep a lot easier tonight after this progress today.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
18 Jan 2024 : Day 142 #
Astonishingly I wake up to find the build I started last night has successfully completed this morning. That's not quite the end of the process though. It means I have a version of gecko implementing a new API. But the API is due to be accessed by qtmozembed. When I rebuild qtmozembed it installs the new gecko packages (which are named "xulrunner-qt5" for historical reasons) and then fails because it can't be built against the new interfaces. 
$ sfdk -c no-fix-version build -d -p
[...]
Loading repository data...
Reading installed packages...
Computing distribution upgrade...
Force resolution: No

The following 5 packages are going to be reinstalled:
  xulrunner-qt5              91.9.1-1
  xulrunner-qt5-debuginfo    91.9.1-1
  xulrunner-qt5-debugsource  91.9.1-1
  xulrunner-qt5-devel        91.9.1-1
  xulrunner-qt5-misc         91.9.1-1

5 packages to reinstall.
[...]
qmozview_p.cpp: In member function ‘void QMozViewPrivate::load(const QString&)’:
qmozview_p.cpp:491:39: error: no matching function for call to
  ‘mozilla::embedlite::EmbedLiteView::LoadURL(char*)’
     mView->LoadURL(url.toUtf8().data());
                                       ^
In file included from qmozview_p.h:26,
                 from qmozview_p.cpp:35:
usr/include/xulrunner-qt5-91.9.1/mozilla/embedlite/EmbedLiteView.h:82:16:
  note: candidate: ‘virtual void mozilla::embedlite::EmbedLiteView::LoadURL
  (const char*, bool)’
   virtual void LoadURL(const char* aUrl, bool aFromExternal);
                ^~~~~~~
usr/include/xulrunner-qt5-91.9.1/mozilla/embedlite/EmbedLiteView.h:82:16:
  note:   candidate expects 2 arguments, 1 provided
This is entirely expected. In fact it's a good sign: it means that things are aligning as intended. The next step will be to update qtmozembed so that it matches the new interface.

The process of fixing this is pretty mechanical: I just have to go through and add the extra fromExternal parameter where it's missing so that it can be passed on directly to EmbedLiteView::LoadURL(). There is one small nuance, which is that on some occasions when the view hasn't been initialised yet, the URL to be loaded is cached until the view is ready. In this case I have to cache the fromExternal state as well, which I'm going to store in QMozViewPrivate::mPendingFromExternal: in the same class as where the URL is cached.

Having made these changes the package now builds successfully. But we're not quite there yet, because these changes are going to cause the exported qtmozembed interfaces to change. These will be picked up by the code in sailfish-browser.

Unlike qtmozembed the changes may not cause sailfish-browser to fail at build time, because it's possible the changes will only affect interpreted QML code rather than compiled C++ code. Let's see...

As I think I've mentioned before, sailfish-browser takes quite a while to build (we're talking tens of minutes rather than hours, but still enough time to make a coffee before it completes). 
$ sfdk -c no-fix-version build -d -p
[...]
Loading repository data...
Reading installed packages...
Computing distribution upgrade...
Force resolution: No

The following 2 packages are going to be reinstalled:
  qtmozembed-qt5        1.53.9-1
  qtmozembed-qt5-devel  1.53.9-1

2 packages to reinstall.
[...]
../../../apps/qtmozembed/declarativewebpage.cpp: In member function
  ‘void DeclarativeWebPage::loadTab(const QString&, bool)’:
../../../apps/qtmozembed/declarativewebpage.cpp:247:20: error: no matching
  function for call to ‘DeclarativeWebPage::load(const QString&)’
         load(newUrl);
                    ^
compilation terminated due to -Wfatal-errors.
As we can see, there are some errors coming from the C++ build relating to the API changes. This is good, but it doesn't negate the fact that some changes will still be needed in the QML as well and these won't be flagged as errors during compilation. So we need to take care, but fixing these C++ errors will be a good start.

Once again the changes look pretty simple; for example here I'm adding the fromExternal parameter: 
void DeclarativeWebPage::loadTab(const QString &newUrl, bool force,
    bool fromExternal)
{
    // Always enable chrome when load is called.
    setChrome(true);
    QString oldUrl = url().toString();
    if ((!newUrl.isEmpty() && oldUrl != newUrl) || force) {
        load(newUrl, fromExternal);
    }
}
These changes have some cascading effects and it takes five or six build cycles to get everything straightened out. But eventually it gets there and the build goes through.

But that's still not quite enough. As we noted earlier this just tackles the compiled code. We can say that the C++ code is probably now in a pretty decent state, but the QML code is a different matter. The build process won't check the QML for inconsistencies and if there are any, it'll just fail at runtime. So we'll need to look through that next. That will have to be a task for tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
17 Jan 2024 : Day 141 #
Preparations are well underway for FOSDEM'24 and I spent this morning putting together a slide template for my talk on Gecko Development. There's not much content yet, but I still have a bit of time to fill that out. It'll be taking up a bit of time between now and then though, so gecko development may slow down a little.

Continuing on though, yesterday we were looking at the Sec-Fetch headers. With the updated browser these are now added as request headers and it turned out this was the reason DuckDuckGo was serving up broken pages. If the headers are skipped a good version of the page is served. But if an ESR 91 user agent is used then the headers are needed and DuckDuckGo expects them to be set to sensible values.

Right now they don't provide sensible values because the context needed isn't available. For example, the browser isn't distinguishing between pages opened at the command line versus those entered as URLs or through clicking on links. We need to pass that info down from the front-end and into the engine.

The flag we need to set in the case of the page being triggered via command line or D-Bus is LOAD_FLAGS_FROM_EXTERNAL: 
  /**
   * A hint this load was prompted by an external program: take care!
   */
  const unsigned long LOAD_FLAGS_FROM_EXTERNAL   = 0x1000;
Once this is set it will end up being used in the following conditional inside SecFetch.cpp: 
  if (!loadInfo->GetLoadTriggeredFromExternal() &&
      !loadInfo->GetHasValidUserGestureActivation()) {
For the GetHasValidUserGestureActivation() portion of this the logic ends up looking like this: 
bool WindowContext::HasValidTransientUserGestureActivation() {
  MOZ_ASSERT(IsInProcess());

  if (GetUserActivationState() != UserActivation::State::FullActivated) {
    MOZ_ASSERT(mUserGestureStart.IsNull(),
               "mUserGestureStart should be null if the document hasn't ever "
               "been activated by user gesture");
    return false;
  }

  MOZ_ASSERT(!mUserGestureStart.IsNull(),
             "mUserGestureStart shouldn't be null if the document has ever "
             "been activated by user gesture");
  TimeDuration timeout = TimeDuration::FromMilliseconds(
      StaticPrefs::dom_user_activation_transient_timeout());

  return timeout <= TimeDuration() ||
         (TimeStamp::Now() - mUserGestureStart) <= timeout;
}
Clearly GetUserActivationState() is critical here. This leads back to WindowContext::NotifyUserGestureActivation() and it looks like this is handled automatically inside the gecko code, so not something to worry about. At least, whether we need to worry about it will become clearer as we progress.

I'm reminded of the fact that back on Day 47 I made amendments to the API to add an aUserActivation parameter to the GoBack() and GoForward() methods of PEmbedLiteView.ipdl and that these probably aren't currently being set properly: 
    async GoBack(bool aRequireUserInteraction, bool aUserActivation);
    async GoForward(bool aRequireUserInteraction, bool aUserActivation);
It's quite possible these will need fixing as well, although right now I can't see any execution path that runs from either of these down to HasValidTransientUserGestureActivation(). This is looking for one path amongst a thousand meandering paths though, so it would be easy to miss it.

Go back to focus on LOAD_FLAGS_FROM_EXTERNAL. To accommodate this I've made some simple changes to EmbedLiteViewChild: 
--- a/embedding/embedlite/embedshared/EmbedLiteViewChild.cpp
+++ b/embedding/embedlite/embedshared/EmbedLiteViewChild.cpp
@@ -478,7 +478,7 @@ EmbedLiteViewChild::WebWidget()
 
 /*----------------------------TabChildIface-----------------------------------------------------*/
 
-mozilla::ipc::IPCResult EmbedLiteViewChild::RecvLoadURL(const nsString &url)
+mozilla::ipc::IPCResult EmbedLiteViewChild::RecvLoadURL(const nsString &url, const bool& aFromExternal)
 {
   LOGT("url:%s", NS_ConvertUTF16toUTF8(url).get());
   NS_ENSURE_TRUE(mWebNavigation, IPC_OK());
@@ -490,6 +490,10 @@ mozilla::ipc::IPCResult EmbedLiteViewChild::RecvLoadURL(const nsString &url)
   }
   flags |= nsIWebNavigation::LOAD_FLAGS_DISALLOW_INHERIT_PRINCIPAL;
 
+  if (aFromExternal) {
+    flags |= nsIWebNavigation::LOAD_FLAGS_FROM_EXTERNAL;
+  }
+
   LoadURIOptions loadURIOptions;
   loadURIOptions.mTriggeringPrincipal = nsContentUtils::GetSystemPrincipal();
   loadURIOptions.mLoadFlags = flags;
This change will have a cascading effect through qtmozembed and potentially all the way to sailfish-browser. But the easiest way for me to figure out what's missing or needs changing is for me to build the package find find out what breaks.

So for the first time in a long time I've set the gecko-dev building once again.

Tomorrow I'll find out what sort of mess I've caused.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
16 Jan 2024 : Day 140 #
I'm up early today in the hope of finding evidence of why DuckDuckGo is serving ESR 91 a completely different — and quite broken — version of the search site. Yesterday we established, pretty convincingly I think, that the page being served is different and that it's not the ESR 91 renderer that's the problem. If I view the exact same pages on ESR 78 or a desktop browser I get the same blank page.

Today I plan to look into the request and response headers, and especially the user agent in more detail. Maybe there's something in the request headers which differs between ESR 78 and ESR 91 that I've not noticed.

But I have to warn you: it's a long one today. I recommend skipping the response headers and backtraces if you're in a rush.

And before getting in to the headers I also want to thank Simon (simonschmeisser) and Adrian (amcewen) for their helpful interjections on the Sailfish Forum and Mastodon respectively.

Both of you correctly noticed that one of the files was broken on the copy I made of DuckDuckGo built from the data downloaded using the ESR 91 library. This shows up quite clearly a desktop browser using the dev console (as in Adrian's picture below) and if you try to download the file directly.
 
A browser developer console showing the filename home-74b6f93f10631d81.js with a very clear 403 error next to it

The broken URL is the following: 
https://duckduckgo.com/_next/static/chunks/pages/%5Blocale%5D/home-74b6f93f10631d81.js
It's broken because of the %5B and %5D in the path. These are characters encoded using "URL" or "percent" encoding as originally codified in RFC 3986. The actually represent the left and right square brackets respectively. What's happened is that the Python script I used back on Day 135 has quite correctly output the URL in this encoded format. When I uploaded it to S3 I should have decoded it so that the file was stored in a folder called [locale] like this: 
https://duckduckgo.com/_next/static/chunks/pages/[locale]/home-74b6f93f10631d81.js
Instead I left the encoded form in. Oh dear! The file itself is tiny, containing just the following text: 
(self.webpackChunk_N_E=self.webpackChunk_N_E||[]).push([[43803],{98387:function
(n,_,u){(window.__NEXT_P=window.__NEXT_P||[]).push(["/[locale]/home",function()
{return u(39265)}])}},function(n){n.O(0,[41966,93432,18040,81125,39337,94623,
95665,55015,61754,55672,38407,49774,92888,40179],(function(){return _=98387,
n(n.s=_);var _}));var _=n.O();_N_E=_}]);
On hearing about my mistake from Simon and Adrian I thought this error would be too small to make any real difference. How wrong I was! Now that I've fixed this the page actually renders now, both in the Sailfish Browser on ESR 78 and on desktop Firefox.

Interestingly it still doesn't render with ESR 91. Which makes me wonder if the issue is related to the locale. But I'm going to have to come back to this because I've committed myself to looking at request headers today. Nevertheless, a big thank you to both Simon and Adrian. Not only is it reassuring to know my work is being checked, but this could also provide a critical piece of the puzzle.

We have a lot to get through today though, so let's now move on to request headers. We're particularly interested in the index.html file because, as anyone who's worked with webservers before will know, this is the first file to get downloaded and the one that kicks off all of the other downloads. Only files referenced in index.html, or where there's a reference chain back to index.html are going to get downloaded.

Here are the request headers for the index page sent to DuckDuckGo when using ESR 78. 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (Mobile; rv:78.0) Gecko/78.0 Firefox/78.0
        Accept : text/html,application/xhtml+xml,application/xml;
            q=0.9,image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
And here are the equivalent headers when accessing the same page of DuckDuckGo when using ESR 91 using the default user agent. 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101
            Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
Finally, when setting the user agent to match that of the iPhone the request headers look like this: 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (iPhone12,1; U; CPU iPhone OS 13_0 like
            Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0
            Mobile/15E148 Safari/602.1
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
What do we notice from these? Obviously the user agents are different. The standard ESR 91 user agent isn't identifying itself as a Mobile variant and that's something that needs to be fixed for all sites. The remaining fields in the ESR 78 list are identical to those in ESR 91. However ESR 91 does have some additional fields: 
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
Let's find out what these do. According to the Mozilla docs, Sec-Fetch-Dest...
 
...allows servers determine whether to service a request based on whether it is appropriate for how it is expected to be used. For example, a request with an audio destination should request audio data, not some other type of resource (for example, a document that includes sensitive user information).

The emphasis here is theirs. In our case we're sending document as the value, which from the docs means:
 
The destination is a document (HTML or XML), and the request is the result of a user-initiated top-level navigation (e.g. resulting from a user clicking a link).

This feels pretty accurate. How about Sec-Fetch-Mode and its value of navigate?
 
Broadly speaking, this allows a server to distinguish between: requests originating from a user navigating between HTML pages, and requests to load images and other resources. For example, this header would contain navigate for top level navigation requests, while no-cors is used for loading an image. For example, this header would contain navigate for top level navigation requests, while no-cors is used for loading an image.

navigate: The request is initiated by navigation between HTML documents.

Again, this all looks pretty unexceptional. Finally the Sec-Fetch-Site header and its value cross-site? According to the docs...
 
...this header tells a server whether a request for a resource is coming from the same origin, the same site, a different site, or is a "user initiated" request. The server can then use this information to decide if the request should be allowed.

cross-site: The request initiator and the server hosting the resource have a different site (i.e. a request by "potentially-evil.com" for a resource at "example.com").

This last one looks suspicious to me. ESR 91 is sending a cross-site value, which is the most risky of all the options, because it's essentially telling DuckDuckGo that the request is happening across different domains. From the docs, it looks like none would be a more appropriate value for this:
 
none: This request is a user-originated operation. For example: entering a URL into the address bar, opening a bookmark, or dragging-and-dropping a file into the browser window.

Checking the code I notice there's also a fourth Sec-Fetch header in the set which is the Sec-Fetch-User header. This isn't being sent and this might also be important, because the request we're making is user-initiated, so it might be reasonable to expect this header to be included. The docs say this about the header:
 
The Sec-Fetch-User fetch metadata request header is only sent for requests initiated by user activation, and its value will always be ?1.

A server can use this header to identify whether a navigation request from a document, iframe, etc., was originated by the user.

In effect, by omitting the header the browser is telling the site that the request isn't initiated by the user, but rather by something else, such as being a document referenced inside some other document.

An obvious thing to test would be to remove these headers and see whether that makes any difference. But before getting on to that let's take a look at the response headers as well.

First the response headers when using ESR 78: 
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Sun, 07 Jan 2024 16:07:57 GMT
        content-type : text/html; charset=UTF-8
        content-length : 2357
        vary : Accept-Encoding
        etag : "65983d82-935"
        content-encoding : br
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : [...] ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Sun, 07 Jan 2024 16:07:56 GMT
        cache-control : no-cache
        X-Firefox-Spdy : h2
I've removed the content-security-policy value for brevity in all of these responses. They all happen to be the same.

Next the response headers when using ESR 91 and the standard user agent: 
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Sun, 07 Jan 2024 16:09:35 GMT
        content-type : text/html; charset=UTF-8
        content-length : 18206
        vary : Accept-Encoding
        etag : "65983d80-471e"
        content-encoding : br
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : [...] ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Sun, 07 Jan 2024 16:09:34 GMT
        cache-control : no-cache
        X-Firefox-Spdy : h2
And finally the response headers when using ESR 91 and the iPhone user agent: 
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Sat, 13 Jan 2024 22:20:49 GMT
        content-type : text/html; charset=UTF-8
        content-length : 18221
        vary : Accept-Encoding
        etag : "65a1788c-472d"
        content-encoding : br
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : [...] ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Sat, 13 Jan 2024 22:20:48 GMT
        cache-control : no-cache
        X-Firefox-Spdy : h2
All three share the same response header keys and values apart from the following three which have different values across all three responses: 
        date : Sun, 07 Jan 2024 16:07:57 GMT
        content-length : 2357
        etag : "65983d82-935"
        expires : Sun, 07 Jan 2024 16:07:56 GMT
It's not surprising these are different across the three. In fact, if they weren't, that might suggest something more sinister. The only one that's really problematic is the content-length value, not because it's incorrect, but because the three different values highlights the fact we're being served three different pages depending on the request.

If there's nothing particularly interesting to see in the response headers it means we can go back to experimenting with the three Sec-Fetch headers discussed earlier.

Digging through the code and the SecFetch.cpp file in particular, I can see that the headers are added in this method: 
void mozilla::dom::SecFetch::AddSecFetchHeader(nsIHttpChannel* aHTTPChannel) {
  // if sec-fetch-* is prefed off, then there is nothing to do
  if (!StaticPrefs::dom_security_secFetch_enabled()) {
    return;
  }

  nsCOMPtr<nsIURI> uri;
  nsresult rv = aHTTPChannel->GetURI(getter_AddRefs(uri));
  if (NS_WARN_IF(NS_FAILED(rv))) {
    return;
  }

  // if we are not dealing with a potentially trustworthy URL, then
  // there is nothing to do here
  if (!nsMixedContentBlocker::IsPotentiallyTrustworthyOrigin(uri)) {
    return;
  }

  AddSecFetchDest(aHTTPChannel);
  AddSecFetchMode(aHTTPChannel);
  AddSecFetchSite(aHTTPChannel);
  AddSecFetchUser(aHTTPChannel);
}
The bit I'm interested in is the first condition which bails out of the method if a specific preference is disabled. Happily this preference was easy to establish as being exposed through about:config as dom.security.secFetch.enabled. I've now disabled it and can try loading the site again.

This time the headers no longer have any of the Sec-Fetch headers included: 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101
            Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
Sadly this doesn't fix the issue, I still get the same blank screen. I try again with the iPhone user agent, same result.

But then I try with the ESR 78 user agent and Sec-Fetch headers disabled... and it works! The site shows correctly, just as it does in ESR 78.

It's a little hard to express the strange mix of jubilation and frustration that I'm feeling right now. Jubilation because we've finally reached the point where it's certain that it will be possible to fix this. Frustration because it's taken quite so long to reach this point.

This feeling pretty much sums up my experience of software development in general. Despite the frustration, this is what I really love about it!

Before claiming that this is fixed, it's worth focusing a little on what the real problem is here. There is a user-agent issue for sure. And arguably DuckDuckGo is trying to frustrate certain users (bots, essentially) by serving different pages to different clients. But the real issue here is that these Sec-Fetch headers are actually broken in our version of gecko ESR 91. That's not the fault of the upstream code: it's a failure of the way the front-end is interacting with the backend code.

So the correct way to fix this issue (at least from the client-side) is to fix those headers. Fixing it for DuckDuckGo is likely to have a positive effect on other sites as well, so fixing it will be worthwhile effort.

That's what I'll now move on to.

As noted above the current (incorrect) values set for the headers are the following: 
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
What we'd expect and want to see for a user-triggered access to DuckDuckGo is this: 
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : none
        Sec-Fetch-User : ?1
Let's check the code for the two incorrect values. First the site value: 
void mozilla::dom::SecFetch::AddSecFetchSite(nsIHttpChannel* aHTTPChannel) {
  nsAutoCString site("same-origin");

  bool isSameOrigin = IsSameOrigin(aHTTPChannel);
  if (!isSameOrigin) {
    bool isSameSite = IsSameSite(aHTTPChannel);
    if (isSameSite) {
      site = "same-site"_ns;
    } else {
      site = "cross-site"_ns;
    }
  }

  if (IsUserTriggeredForSecFetchSite(aHTTPChannel)) {
    site = "none"_ns;
  }

  nsresult rv =
      aHTTPChannel->SetRequestHeader("Sec-Fetch-Site"_ns, site, false);
  mozilla::Unused << NS_WARN_IF(NS_FAILED(rv));
}
We can infer that isSameOrigin and isSameSite are both set to false. This is a bit strange but it's actually not the bit we have to worry about. The result of going through the initial condition will be overwritten if IsUserTriggeredForSecFetchSite() returns true so that's where we should focus.

I'm going to use the debugger to try to find out how these values get set. 
$ rm -rf ~/.local/share/org.sailfishos/browser/.mozilla/cache2/ \
    ~/.local/share/org.sailfishos/browser/.mozilla/startupCache/ ~/.local/share/org.sailfishos/browser/.mozilla/cookies.sqlite 
$ gdb sailfish-browser
[...]
(gdb) r
[...]

(gdb) b LoadInfo::SetHasValidUserGestureActivation
Breakpoint 4 at 0x7fb9d34424: file netwerk/base/LoadInfo.cpp, line 1609.
(gdb) b LoadInfo::SetLoadTriggeredFromExternal
Breakpoint 5 at 0x7fb9d34300: file netwerk/base/LoadInfo.cpp, line 1472.
(gdb) c

Thread 8 "GeckoWorkerThre" hit Breakpoint 4, mozilla::net::LoadInfo::
    SetHasValidUserGestureActivation (this=this@entry=0x7f89b00da0, 
    aHasValidUserGestureActivation=false) at netwerk/base/LoadInfo.cpp:1609
1609      mHasValidUserGestureActivation = aHasValidUserGestureActivation;
(gdb) bt
#0  mozilla::net::LoadInfo::SetHasValidUserGestureActivation
    (this=this@entry=0x7f89b00da0, aHasValidUserGestureActivation=false)
    at netwerk/base/LoadInfo.cpp:1609
#1  0x0000007fb9ffd86c in mozilla::net::CreateDocumentLoadInfo
    (aBrowsingContext=aBrowsingContext@entry=0x7f88c6dde0, 
    aLoadState=aLoadState@entry=0x7f894082d0)
    at netwerk/ipc/DocumentLoadListener.cpp:149
#2  0x0000007fb9ffd9b8 in mozilla::net::DocumentLoadListener::OpenDocument
    (this=0x7f89a8c3e0, aLoadState=0x7f894082d0, aCacheKey=0, aChannelId=..., 
    aAsyncOpenTime=..., aTiming=0x0, aInfo=..., aUriModified=...,
    aIsXFOError=..., aPid=aPid@entry=0, aRv=aRv@entry=0x7f9f3eda34)
    at netwerk/ipc/DocumentLoadListener.cpp:744
#3  0x0000007fb9ffe88c in mozilla::net::ParentProcessDocumentChannel::AsyncOpen
    (this=0x7f89bf02e0, aListener=0x7f895a7770)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/MaybeStorageBase.h:80
#4  0x0000007fba4663bc in nsURILoader::OpenURI (this=0x7f887b46c0,
    channel=0x7f89bf02e0, aFlags=0, aWindowContext=0x7f886044f0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#5  0x0000007fbc7fa824 in nsDocShell::OpenInitializedChannel
    (this=this@entry=0x7f886044c0, aChannel=0x7f89bf02e0,
    aURILoader=0x7f887b46c0, aOpenFlags=0)
    at docshell/base/nsDocShell.cpp:10488
#6  0x0000007fbc7fb5e4 in nsDocShell::DoURILoad (this=this@entry=0x7f886044c0,
    aLoadState=aLoadState@entry=0x7f894082d0, aCacheKey=..., 
    aRequest=aRequest@entry=0x7f9f3edf90)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#7  0x0000007fbc7fc1a4 in nsDocShell::InternalLoad
    (this=this@entry=0x7f886044c0, aLoadState=aLoadState@entry=0x7f894082d0,
    aCacheKey=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:1363
#8  0x0000007fbc801c20 in nsDocShell::ReloadDocument
    (aDocShell=aDocShell@entry=0x7f886044c0, aDocument=<optimized out>,
    aLoadType=aLoadType@entry=2, aBrowsingContext=0x7f88c6dde0,
    aCurrentURI=0x7f887e62c0, aReferrerInfo=0x0,
    aNotifiedBeforeUnloadListeners=aNotifiedBeforeUnloadListeners@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/MaybeStorageBase.h:79
#9  0x0000007fbc803984 in nsDocShell::Reload (this=0x7f886044c0, aReloadFlags=0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#10 0x0000007fbc94d05c in nsWebBrowser::Reload (this=<optimized out>,
    aReloadFlags=<optimized out>)
    at toolkit/components/browser/nsWebBrowser.cpp:507
#11 0x0000007fbcb0ab70 in mozilla::embedlite::EmbedLiteViewChild::RecvReload
    (this=<optimized out>, aHardReload=<optimized out>)
    at mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:533
#12 0x0000007fba1915d0 in mozilla::embedlite::PEmbedLiteViewChild::
    OnMessageReceived (this=0x7f88767020, msg__=...)
    at PEmbedLiteViewChild.cpp:1152
#13 0x0000007fba17f05c in mozilla::embedlite::PEmbedLiteAppChild::
    OnMessageReceived (this=<optimized out>, msg__=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:675
#14 0x0000007fba06b85c in mozilla::ipc::MessageChannel::DispatchAsyncMessage
    (this=this@entry=0x7f88b3e8a8, aProxy=aProxy@entry=0x7ebc003140, aMsg=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:675
[...]
#40 0x0000007fb78b289c in ?? () from /lib64/libc.so.6
(gdb) c
Continuing.

Thread 8 "GeckoWorkerThre" hit Breakpoint 5, mozilla::net::LoadInfo::
    SetLoadTriggeredFromExternal (this=this@entry=0x7f89b00da0, 
    aLoadTriggeredFromExternal=false) at netwerk/base/LoadInfo.cpp:1472
1472      mLoadTriggeredFromExternal = aLoadTriggeredFromExternal;
(gdb) bt
#0  mozilla::net::LoadInfo::SetLoadTriggeredFromExternal
    (this=this@entry=0x7f89b00da0, aLoadTriggeredFromExternal=false)
    at netwerk/base/LoadInfo.cpp:1472
#1  0x0000007fbc7f20f8 in nsDocShell::CreateAndConfigureRealChannelForLoadState
    (aBrowsingContext=aBrowsingContext@entry=0x7f88c6dde0, 
    aLoadState=aLoadState@entry=0x7f894082d0,
    aLoadInfo=aLoadInfo@entry=0x7f89b00da0,
    aCallbacks=aCallbacks@entry=0x7f89724110, 
    aDocShell=aDocShell@entry=0x0, aOriginAttributes=...,
    aLoadFlags=aLoadFlags@entry=2689028, aCacheKey=aCacheKey@entry=0, 
    aRv=@0x7f9f3eda34: nsresult::NS_OK, aChannel=aChannel@entry=0x7f89a8c438)
    at docshell/base/nsDocShellLoadState.cpp:709
#2  0x0000007fb9ff9c4c in mozilla::net::DocumentLoadListener::Open
    (this=this@entry=0x7f89a8c3e0, aLoadState=aLoadState@entry=0x7f894082d0, 
    aLoadInfo=aLoadInfo@entry=0x7f89b00da0, aLoadFlags=2689028,
    aCacheKey=aCacheKey@entry=0, aChannelId=..., aAsyncOpenTime=..., 
    aTiming=aTiming@entry=0x0, aInfo=..., aUrgentStart=aUrgentStart@entry=false,
    aPid=aPid@entry=0, aRv=aRv@entry=0x7f9f3eda34)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:359
#3  0x0000007fb9ffda34 in mozilla::net::DocumentLoadListener::OpenDocument
    (this=0x7f89a8c3e0, aLoadState=0x7f894082d0, aCacheKey=0, aChannelId=..., 
    aAsyncOpenTime=..., aTiming=0x0, aInfo=..., aUriModified=...,
    aIsXFOError=..., aPid=aPid@entry=0, aRv=aRv@entry=0x7f9f3eda34)
    at netwerk/ipc/DocumentLoadListener.cpp:750
#4  0x0000007fb9ffe88c in mozilla::net::ParentProcessDocumentChannel::AsyncOpen
    (this=0x7f89bf02e0, aListener=0x7f895a7770)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/MaybeStorageBase.h:80
#5  0x0000007fba4663bc in nsURILoader::OpenURI (this=0x7f887b46c0,
    channel=0x7f89bf02e0, aFlags=0, aWindowContext=0x7f886044f0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#6  0x0000007fbc7fa824 in nsDocShell::OpenInitializedChannel
    (this=this@entry=0x7f886044c0, aChannel=0x7f89bf02e0,
    aURILoader=0x7f887b46c0, aOpenFlags=0)
    at docshell/base/nsDocShell.cpp:10488
#7  0x0000007fbc7fb5e4 in nsDocShell::DoURILoad (this=this@entry=0x7f886044c0,
    aLoadState=aLoadState@entry=0x7f894082d0, aCacheKey=..., 
    aRequest=aRequest@entry=0x7f9f3edf90)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#8  0x0000007fbc7fc1a4 in nsDocShell::InternalLoad (this=this@entry=0x7f886044c0,
    aLoadState=aLoadState@entry=0x7f894082d0, aCacheKey=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:1363
#9  0x0000007fbc801c20 in nsDocShell::ReloadDocument
    (aDocShell=aDocShell@entry=0x7f886044c0, aDocument=<optimized out>,
    aLoadType=aLoadType@entry=2, aBrowsingContext=0x7f88c6dde0,
    aCurrentURI=0x7f887e62c0, aReferrerInfo=0x0,
    aNotifiedBeforeUnloadListeners=aNotifiedBeforeUnloadListeners@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/MaybeStorageBase.h:79
#10 0x0000007fbc803984 in nsDocShell::Reload (this=0x7f886044c0, aReloadFlags=0)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsCOMPtr.h:859
#11 0x0000007fbc94d05c in nsWebBrowser::Reload (this=<optimized out>,
    aReloadFlags=<optimized out>)
    at toolkit/components/browser/nsWebBrowser.cpp:507
#12 0x0000007fbcb0ab70 in mozilla::embedlite::EmbedLiteViewChild::RecvReload
    (this=<optimized out>, aHardReload=<optimized out>)
    at mobile/sailfishos/embedshared/EmbedLiteViewChild.cpp:533
#13 0x0000007fba1915d0 in mozilla::embedlite::PEmbedLiteViewChild::
    OnMessageReceived (this=0x7f88767020, msg__=...)
    at PEmbedLiteViewChild.cpp:1152
#14 0x0000007fba17f05c in mozilla::embedlite::PEmbedLiteAppChild::
    OnMessageReceived (this=<optimized out>, msg__=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:675
#15 0x0000007fba06b85c in mozilla::ipc::MessageChannel::DispatchAsyncMessage
    (this=this@entry=0x7f88b3e8a8, aProxy=aProxy@entry=0x7ebc003140, aMsg=...)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/ipc/ProtocolUtils.h:675
[...]
#41 0x0000007fb78b289c in ?? () from /lib64/libc.so.6
(gdb) c
[...]
Digging deeper to find out where the value for mHasValidUserGestureActivation is coming from. It's being set to false and we need to know why. 
(gdb) disable break
(gdb) break nsDocShell.cpp:4108
Breakpoint 6 at 0x7fbc801bec: file docshell/base/nsDocShell.cpp, line 4108.
(gdb) c
Continuing.
[Switching to LWP 29933]

Thread 8 "GeckoWorkerThre" hit Breakpoint 6, nsDocShell::ReloadDocument
    (aDocShell=aDocShell@entry=0x7f886044c0, aDocument=<optimized out>, 
    aLoadType=aLoadType@entry=2, aBrowsingContext=0x7f88c6dde0,
    aCurrentURI=0x7f887e62c0, aReferrerInfo=0x0, 
    aNotifiedBeforeUnloadListeners=aNotifiedBeforeUnloadListeners@entry=false)
    at docshell/base/nsDocShell.cpp:4108
4108      loadState->SetHasValidUserGestureActivation(
(gdb) p context
$14 = {mRawPtr = 0x7ed4009980}
(gdb) p context.mRawPtr
$15 = (mozilla::dom::WindowContext *) 0x7ed4009980
(gdb) p *(context.mRawPtr)
$16 = {<nsISupports> = {_vptr.nsISupports = 0x7fbf75c7a0 <vtable for
    mozilla::dom::WindowGlobalParent+16>}, <nsWrapperCache> = {
[...]
    mInnerWindowId = 16, mOuterWindowId = 1, mBrowsingContext = { mRawPtr =
    0x7f88c6dde0}, mWindowGlobalChild = {mRef = {mRawPtr = 0x7f895aa790}}, 
  mChildren = {<nsTArray_Impl<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArrayInfallibleAllocator>> = {
      <nsTArray_base<nsTArrayInfallibleAllocator,
      nsTArray_RelocateUsingMemutils>> = { mHdr = 0x7fbe0c86b8
      <sEmptyTArrayHeader>},
      <nsTArray_TypedBase<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArray_Impl<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArrayInfallibleAllocator> >> = {
      <nsTArray_SafeElementAtHelper<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArray_Impl<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArrayInfallibleAllocator> >> =
      {<nsTArray_SafeElementAtSmartPtrHelper<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArray_Impl<RefPtr<mozilla::dom::BrowsingContext>,
      nsTArrayInfallibleAllocator> >> = {<detail::nsTArray_CopyDisabler> =
      {<No data fields>}, <No data fields>}, <No data fields>},
      <No data fields>}, static NoIndex = 18446744073709551615},
      <No data fields>}, mIsDiscarded = false, mIsInProcess = true,
      mCanExecuteScripts = true, 
  mUserGestureStart = {mValue = {mUsedCanonicalNow = 0, mTimeStamp = 0}}}
(gdb)
I eventually end up here in nsDocShell.cpp: 
  aLoadInfo->SetLoadTriggeredFromExternal(
      aLoadState->HasLoadFlags(LOAD_FLAGS_FROM_EXTERNAL));
This is important because of external are considered user triggered as mentioned in the comments in SecFetch::AddSecFetchUser(): 
  // sec-fetch-user only applies if the request is user triggered.
  // requests triggered by an external application are considerd user triggered.
  if (!loadInfo->GetLoadTriggeredFromExternal() &&
      !loadInfo->GetHasValidUserGestureActivation()) {
    return;
  }
These load flags are set all over the place, although the LOAD_FLAGS_FROM_EXTERNAL specifically is only set in tabbrowser.js in the upstream code, which I believe is code we don't use for the Sailfish Browser. Instead we set these flags in EmbedLiteViewChild.cpp. It's possible there are some new flags which we need to add there. Let's check the history using git blame. I happen to see from the source code that the flag is being set on line 1798 of the tabbrowser.js file: 
$ git blame browser/base/content/tabbrowser.js -L1798,1798
f9f59140398bc (Victor Porof 2019-07-05 09:48:57 +0200 1798)
    flags |= Ci.nsIWebNavigation.LOAD_FLAGS_FROM_EXTERNAL;
$ git log -1 --oneline f9f59140398bc
f9f59140398b Bug 1561435 - Format browser/base/, a=automatic-formatting
$ git log -1 f9f59140398bc
commit f9f59140398bc4d04d840e8217c04e0d7eafafb9
Author: Victor Porof <vporof@mozilla.com>
Date:   Fri Jul 5 09:48:57 2019 +0200

    Bug 1561435 - Format browser/base/, a=automatic-formatting
    
    # ignore-this-changeset
    
    Differential Revision: https://phabricator.services.mozilla.com/D36041
    
    --HG--
    extra : source : 96b3895a3b2aa2fcb064c85ec5857b7216884556
This commit is just reformatting the file so we need to look at the change prior to this one. Checking the diff of the automatic formatting I can see that before this the relevant line of code was line 1541 in the same file. 
$ git blame f9f59140398bc~ browser/base/content/tabbrowser.js -L1541,1541
082b6eb1e7ed2 (James Willcox 2019-03-12 20:20:58 +0000 1541)
    flags |= Ci.nsIWebNavigation.LOAD_FLAGS_FROM_EXTERNAL;
$ git log -1 082b6eb1e7ed2
commit 082b6eb1e7ed20de7424aea94fb7ce40b1b39c36
Author: James Willcox <snorp@snorp.net>
Date:   Tue Mar 12 20:20:58 2019 +0000

    Bug 1524992 - Treat command line URIs as external r=mconley
    
    Differential Revision: https://phabricator.services.mozilla.com/D20890
    
    --HG--
    extra : moz-landing-system : lando
This is the important change. It added the flag specifically for URIs triggered at the command line. As it happens that's currently one of the ways I've been testing, so I should fix this. It's worth noting that this flag was introduced in ESR 67 so has actually been around for a while. But I guess skipping it didn't have any obvious negative effects, so nobody noticed it needed to be handled.

That'll have to change now. But it looks like this will be just one of many such flags that will need adding in to the Sailfish code.

Let's focus on LOAD_FLAGS_FROM_EXTERNAL first. This is set in browser.js when a call is made to getContentWindowOrOpenURI(). This ends up running this piece of code: 
      default:
        // OPEN_CURRENTWINDOW or an illegal value
        browsingContext = window.gBrowser.selectedBrowser.browsingContext;
        if (aURI) {
          let loadFlags = Ci.nsIWebNavigation.LOAD_FLAGS_NONE;
          if (isExternal) {
            loadFlags |= Ci.nsIWebNavigation.LOAD_FLAGS_FROM_EXTERNAL;
          } else if (!aTriggeringPrincipal.isSystemPrincipal) {
            // XXX this code must be reviewed and changed when bug 1616353
            // lands.
            loadFlags |= Ci.nsIWebNavigation.LOAD_FLAGS_FIRST_LOAD;
          }
          gBrowser.loadURI(aURI.spec, {
            triggeringPrincipal: aTriggeringPrincipal,
            csp: aCsp,
            loadFlags,
            referrerInfo,
          });
        }
We end up here when there's a call made to handURIToExistingBrowser() in BrowserContentHandler.jsm which then calls browserDOMWindow.openURI() with the aFlags parameter set to Ci.nsIBrowserDOMWindow.OPEN_EXTERNAL.

What's the equivalent for Sailfish Browser? There are three ways it may end up opening an external URL that I can think of. The first is if the application is executed with a URL on the command line: 
$ sailfish-browser https://www.flypig.co.uk/gecko
Another is if xdg-open is called: 
$ xdg-open https://www.flypig.co.uk/gecko
The third is if the browser is called via its D-Bus interface: 
$ gdbus call --session --dest org.sailfishos.browser.ui --object-path /ui \
    --method org.sailfishos.browser.ui.openUrl \
    "['https://www.flypig.co.uk/gecko']"
All of these should end up setting the LOAD_FLAGS_FROM_EXTERNAL flag.

Let's follow the path in the case of the D-Bus call. The entry point for this is in browserservice.cpp where the D-Bus object is registered. The call looks like this: 
void BrowserUIService::openUrl(const QStringList &args)
{
    if(args.count() > 0) {
        emit openUrlRequested(args.first());
    } else {
        emit openUrlRequested(QString());
    }
}
This gets picked up by the browser object via a connection in main.cpp: 
        browser->connect(uiService, &BrowserUIService::openUrlRequested,
                        browser, &Browser::openUrl);
Which triggers the following method: 
void Browser::openUrl(const QString &url)
{
    Q_D(Browser);
    DeclarativeWebUtils::instance()->openUrl(url);
}
The version of the method in DeclarativeWebUtils sanitises the url before sending it on its way by emitting a signal: 
    emit openUrlRequested(tmpUrl);
Finally this is picked up by BrowserPage.qml which ends up doing one of three things with it: 
    Connections {
        target: WebUtils
        onOpenUrlRequested: {
[...]
            if (webView.tabModel.activateTab(url)) {
                webView.releaseActiveTabOwnership()
            } else if (!webView.tabModel.loaded) {
                webView.load(url)
            } else {
                webView.clearSelection()
                webView.tabModel.newTab(url)
                overlay.dismiss(true, !Qt.application.active /* immadiate */)
            }
So it either activates an existing tab, calls the user interface to create the very first tab, or adds a new tab to the existing list. This approach contrasts with the route taken when the user enters a URL. In this case the process is handled in Overlay.qml and the loadPage() function there. This does a bunch of checks before calling this: 
                webView.load(pageUrl)
Notice that this is also one of the methods that's called in the case of the D-Bus trigger as well. That's important, because we need to distinguish between these two routes. The WebView component inherits load() from DeclarativeWebContainer where the method looks like this: 
void DeclarativeWebContainer::load(const QString &url, bool force)
{
    QString tmpUrl = url;
    if (tmpUrl.isEmpty() || !browserEnabled()) {
        tmpUrl = ABOUT_BLANK;
    }

    if (!canInitialize()) {
        m_initialUrl = tmpUrl;
    } else if (m_webPage && m_webPage->completed()) {
        if (m_loading) {
            m_webPage->stop();
        }
        m_webPage->loadTab(tmpUrl, force);
        Tab *tab = m_model->getTab(m_webPage->tabId());
        if (tab) {
            tab->setRequestedUrl(tmpUrl);
        }
    } else if (m_model && m_model->count() == 0) {
        // Browser running all tabs are closed.
        m_model->newTab(tmpUrl);
    }
}
Notice that this mimics the D-Bus route with there being three options: record the URL in case there's a failure, load the URL into an existing tab or create a new tab if there are none already available.

Trying to follow all these routes is like trying to follow a droplet of water down a waterfall. I think I've reached the limit of my indirection capacity here; I'm going to need to pick this up again tomorrow.

But, so that I don't lose my thread, a note about two further things I need to do with this.

First I need to follow the process through until I hit the point at which the EmbedLiteViewParent::SendLoadURL() is called. This is the point at which the flag needs to be set. It looks like the common way for this to get called is through the following call: 
void
EmbedLiteView::LoadURL(const char* aUrl)
{
  LOGT("url:%s", aUrl);
  Unused << mViewParent->SendLoadURL(NS_ConvertUTF8toUTF16(nsDependentCString(aUrl)));
}
I should check this with the debugger to make sure.

Second I need to ensure the flag gets passed from the point at which we know what its value needs to be (which is the D-Bus interface) to this call to SendLoadURL(), so that EmbedLiteViewChild::RecvLoadURL() can set it appropriately.

Once I have those two pieces everything will tie together and at that point it will be possible to set the LOAD_FLAGS_FROM_EXTERNAL flag appropriately.

That's it for today. Tomorrow, onwards.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
15 Jan 2024 : Day 139 #
This morning I transferred the DuckDuckGo testing website from S3 where we had it yesterday to Amazon CloudFront. This still uses the S3 bucket to serve the pages as a static site, but now with a cloudfront.net URL (which shouldn't make any difference for the tests) and using HTTPS (which might make a difference). I want to use HTTPS not because I'm worried about the integrity of the site, but because I want to eliminate differences between how I'm accessing the real DuckDuckGo and the test page version I'm using.

A small, overly-optimistic, bit of me was hoping that the switch to HTTPS might cause the site to break when using ESR 91, but it didn't. The DuckDuckGo logo shows just fine, the page looks sensible, albeit still showing the peculiar differences in functionality that I described yesterday.

The next steps are to capture the full output from accessing the original site using ESR 91, followed by the same process accessing the test site using ESR 91. In order to do this I'll need to build and run the recent changes I made to the EmbedLiteConsoleListener.js code so that it can be installed on my ESR 91 device. I made the changes manually on my ESR 78 device, and although I did already manually copy those changes over to the local repository on my development machine, I've not yet built and installed them for my other phone.

Having done that, I now need to capture a copy of the site using the ESR 91 version of the browser. I'll capture both a copy of the real site and a copy of the replicated version of the site (the version downloaded using ESR 78): 
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser \
    https://d3gf5xld99gmbj.cloudfront.net/ 2>&1 > esr91-ddg-copy.txt
[...]
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser \
    https://duckduckgo.com/ 2>&1 > esr91-ddg-orig.txt
[...]
Looking through the output of both it's clear that they're quite different. Just starting with the index.html page, the very first line of each differ significantly. So it really does seem to be the case that DuckDuckGo is serving a different version of the page.

I also tried downloading a copy using ESR 91 and the iPhone user agent string from yesterday. But the site downloaded was the same.

What I want to do now is create a copy of the site downloaded when I use ESR 91. This is the site that's broken (showing just a blank page) when rendered using the ESR 91 renderer. But although the page is blank it is still downloading a bunch of files, so there's definitely something to replicate.

Having done this process before I'm getting quite proficient at it now. The process goes like this:
  1. Take the log output from loading the page, with the full network dump enabled
  2. Work through this log file and whenever there's some text content in the log, cut and paste this out into its own file. This is then a copy of the file as it was downloaded by the Sailfish Browser.
  3. Carefully save this file out using the same file structure as served by the server (matching the suffix of the URL of the file downloaded).
  4. Having recreated all of these files, create an S3 bucket on AWS and copy all of these files in to it.
  5. Create a CloudFront distribution of the bucket. To reiterate, I do it this way rather than just serving the bucket as a static site so that I can offer the site with HTTPS access.
I've been through all of these steps again and am now using CloudFront to serve two copies of the site:
  1. ddg8: the original DuckDuckGo site as downloaded by ESR 78: https://d3gf5xld99gmbj.cloudfront.net/.
  2. ddg9: the original DuckDuckGo site as downloaded by ESR 91: https://dd53jyxmgchu8.cloudfront.net/.
If you take a look at these sites you'll see that the version downloaded using ESR 78 looks pretty decent when downloaded by using a desktop browser. But the one downloaded by ESR 91 is blank, just as it is when rendering it on Sailfish OS using ESR 91.

There's one final check to make and that's to access the copy of the site originally downloaded using ESR 91 and now being served on CloudFront (the ddg9 version of the site), but using ESR 91. Why do this? Once I've got this I can compare the files downloaded this way with the files downloaded directly from DuckDuckGo using ESR 91.

If the copy is good, the files downloaded should be very similar, if not identical.

I've done this now, so have two copies of the log output. Let's compare them using the comparison command I put together a few days back. 
$ diff --side-by-side 
  <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" esr91-ddg-orig-urls.txt) | sort) \
  <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "cloudfront" esr91-ddg-esr91-urls.txt) | sort)

18040-1287342b1f839f70.js                    18040-1287342b1f839f70.js
38407-070351ade350c8e4.js                    38407-070351ade350c8e4.js
39337-cd8caeeff0afb1c4.js                    39337-cd8caeeff0afb1c4.js
41966-c9d76895b4f9358f.js                    41966-c9d76895b4f9358f.js
55015-29fec414530c2cf6.js                    55015-29fec414530c2cf6.js
55672-0a2c33e517ba92f5.js                    55672-0a2c33e517ba92f5.js
61754-cfebc3ba4c97208e.js                    61754-cfebc3ba4c97208e.js
6a4833195509cc3d.css                         6a4833195509cc3d.css
703c9a9a057785a9.css                         703c9a9a057785a9.css
81125-b74d1b6f4908497b.js                    81125-b74d1b6f4908497b.js
93432-ebd443fe69061b19.js                    93432-ebd443fe69061b19.js
94623-d5bfa67fc3bada59.js                    94623-d5bfa67fc3bada59.js
95665-f2a003aa56f899b0.js                    95665-f2a003aa56f899b0.js
a2a29f84956f2aac.css                         a2a29f84956f2aac.css
_app-a1aac13e30ca1ed6.js                     _app-a1aac13e30ca1ed6.js
_buildManifest.js                            _buildManifest.js
c89114cfe55133c4.css                         c89114cfe55133c4.css
ed8494aa71104fdc.css                         ed8494aa71104fdc.css
f0b3f7da285c9dbd.css                         f0b3f7da285c9dbd.css
framework-f8115f7fae64930e.js                framework-f8115f7fae64930e.js
home-34dda07336cb6ee1.js                     home-34dda07336cb6ee1.js
main-17a05b704438cdd6.js                     main-17a05b704438cdd6.js
ProximaNova-Bold-webfont.woff2               ProximaNova-Bold-webfont.woff2
ProximaNova-ExtraBold-webfont.woff2          ProximaNova-ExtraBold-webfont.woff2
ProximaNova-RegIt-webfont.woff2              ProximaNova-RegIt-webfont.woff2
ProximaNova-Reg-webfont.woff2                ProximaNova-Reg-webfont.woff2
ProximaNova-Sbold-webfont.woff2              ProximaNova-Sbold-webfont.woff2
_ssgManifest.js                              _ssgManifest.js
webpack-96503cdd116848e8.js                  webpack-96503cdd116848e8.js
The two sets of downloaded files are identical. This is really good, because it means that the ddg9 version of the site is an accurate reflection of what's being served to ESR 91 when Sailfish Browser accesses the real DuckDuckGo site using the ESR 91 engine.

Visiting this copy of the site in other browsers, including ESR 78 and the desktop version of Firefox, shows that the site doesn't render there either.

It's been a long journey to get to this point, but this is clear evidence that the problem is the site being served to ESR 91, rather than the ESR 91 rendering engine or JavaScript interpreter getting stuck after the site has been received.

This means I have to concentrate on persuading DuckDuckGo to serve the same version of the page that it's serving to ESR 78. It's taken far too long to get here, but at least I feel I've learnt something in the process about how to perform these checks, how to download accurate copies of a website and how to serve them using CloudFront so that they don't have to go in a sub-directory.

I've already tried with multiple different user agents and that hasn't been enough to persuade DuckDuckGo to serve the correct version of the page, so I'm not quite sure how to get around this issue. One possibility is that I'm not actually using the user agents I think I am. So this will be something to check tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
14 Jan 2024 : Day 138 #
Finally, after much meandering and misdirected effort, an important reason for my copied version of DuckDuckGo failing became clear yesterday. I admit I'm a bit embarrassed that I didn't spot it earlier, but there's no hiding here.

Simply put, DuckDuckGo uses paths with a preceding / in the site HTML file rather than fully relative URLs. So by placing the files in a "tests/ddg8" folder on my server, I was breaking most of the links.

Now, admittedly, I've not yet had a chance to see what happens when this is fixed, so there could well be other issues as well. But what's for sure is that without fixing this, the copied site will remain broken.

My plan is to make use of the HTML base element to try to work around the issue. This can be added to the head of an HTML file to direct the browser to the root of the site, so that all relative URLs are resolved relative the the base address.

I should also check for URLs that start with https://duckduckgo.com/ or similar as these won't be fixed by this change.

Since the location of my test site is https://www.flypig.co.uk/tests/ddg8/ the addition I need to make is a line inside the head element of the index.html file like this: 
   <base href="https://www.flypig.co.uk/tests/ddg8/" />
On trying this, in practice and contrary to what I'd expected, it turns out that when a URL has a preceding / it's considered an absolute URL as well. The spec wasn't clear on this point for me, but it means it's resolved relative to the domain name, not relative to the base. That's rubbish, but observable behaviour. Rubbish because it means I can't use this as my solution after all.

So I'm going to have to make more intrusive changes, removing these preceding slashes from all instances of the URL in the page and all files that get loaded with it. I was hoping to avoid that.

There are alternatives to this intrusive fix though. Here are the three alternatives I can think of:
  1. Move the site to the root of the URL. This will get it mixed up with the rest of my site so I'd rather not do that.
  2. Move it to a completely new URL. This is definitely an option. I could spin up a Cloud server for this pretty easily.
  3. Configure the server so that it serves the directory from the root URL. The would be cheaper than using a Cloud service.
To save myself some time and effort I've decided to go with the second option. I don't think I'll need to keep the site up for long so the cost will be minimal and it'll prevent me causing a mess. Once I'm done I can shutdown the site and everything will be as before.

So this is what I've done. I didn't end up spinning up a server but rather copied the files over to an S3 bucket no Amazon Web Services. There's an option to serve static files from an S3 bucket as a website, which is exactly what I need.
 
Using AWS S3 to serve a test copy of the DuckDuckGo site

Testing the site using desktop Firefox shows a much better result than before. It's not perfect: there are still some missing images, but the copy I made to the bucket is the mobile version, so that's to be expected. Nevertheless it makes for a pretty reasonable facsimile of the real DuckDuckGo site.
 
The DDG test site rendered using a desktop browser

But what about if I try it on mobile?

Using ESR 78 it's uncannily similar to the real DuckDuckGo site. Even the menu on the left hand side works. Search suggestions and search itself are of course both broken, but again this is entirely to be expected since these features require dynamic aspects of the site which can't be replicated using an S3 bucket.
 
The same DDG test site rendered using ESR 78 (left) and ESR 91 (right) renderers. They're almost identical, apart from the downward scroll arrow which only shows on ESR 78.

But the real test is ESR 91. When I try it there... I also get a good rendition of the DuckDuckGo site. This is both good and bad. If it had failed to render that would have been ideal, because then I'd immediately have a comparison to work with. But the fact that it works well means I can now compare it with the real version of the site and try to figure out what's different.

It's also worth noting that the results aren't the same. On ESR 78 I can scroll down to view all the "bathroomguy" images telling me how great the site is. On ESR 91 the rendered down arrow is missing and I'm not able to scroll, as you can see in the screenshots if you peer closely.

So, what's the difference? That'll be my task for tomorrow!

This lends more weight to the claim that the problem here DuckDuckGo serving different pages rather than the ESR 91 renderer or JavaScript engine choking on something. I have a plan for how to test this categorically tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
13 Jan 2024 : Day 137 #
I'm still trying to get myself a copy of the DuckDuckGo website. To recap the latest situation, I still want a copy I can serve from my personal server, but which triggers the same errors as when accessing DuckDuckGo from the real website using ESR 91.

I feel confident that yesterday I got myself a full verbatim copy of the site. The catch is that it's woven into the logging output from ESR 91. My task today is to disentangle it.

The log file is 1.3 MiB of data. That's not crazy quantities, but it would work better if the text files used line-wrapping, rather than including massively long lines that seemingly go on for ever... my text editor really doesn't like having to deal with them and just hangs up for tens of minutes at a time.

[...]

Nevertheless, and although it took an age, I have managed to get it all done. The file structure is very similar to the one I showed yesterday: 
$ tree ddg8/
ddg8/
├── assets
│   ├── logo_homepage.alt.v109.svg
│   ├── logo_homepage.normal.v109.svg
│   └── onboarding
│       ├── arrow.svg
│       └── bathroomguy
│           ├── 1-monster-v2--no-animation.svg
│           ├── 2-ghost-v2.svg
│           ├── 3-bathtub-v2--no-animation.svg
│           ├── 4-alpinist-v2.svg
│           └── teaser-2@2x.png
├── dist
│   ├── b.9e45618547aaad15b744.js
│   ├── d.01ff355796b8725c8dad.js
│   ├── h.2d6522d4f29f5b108aed.js
│   ├── lib
│   │   └── l.656ceb337d61e6c36064.js
│   ├── o.2988a52fdfb14b7eff16.css
│   ├── p.f5b58579149e7488209f.js
│   ├── s.b49dcfb5899df4f917ee.css
│   ├── ti.b07012e30f6971ff71d3.js
│   ├── tl.3db2557c9f124f3ebf92.js
│   └── util
│       └── u.a3c3a6d4d7bf9244744d.js
├── font
│   ├── ProximaNova-ExtraBold-webfont.woff2
│   ├── ProximaNova-Reg-webfont.woff2
│   └── ProximaNova-Sbold-webfont.woff2
├── index.html
├── locale
│   └── en_GB
│       └── duckduckgo85.js
└── post3.html

9 directories, 24 files
And not just that, but in fact the contents are very similar overall: 
$ diff -q ddg ddg8/
Only in ddg: 3.html
Common subdirectories: ddg/assets and ddg8/assets
Common subdirectories: ddg/dist and ddg8/dist
Common subdirectories: ddg/font and ddg8/font
Files ddg/index.html and ddg8/index.html differ
Common subdirectories: ddg/locale and ddg8/locale
Only in ddg8/: post3.html
Although the index.html file is quite different to the equivalent one I downloaded earlier, it is similar to a previous one that was downloaded using the python script: 
$ diff ddg5/index.html ddg8/index.html 
2,7c2,7
< <!--[if IEMobile 7 ]> <html lang="en-US" class="no-js iem7"> <![endif]-->
< <!--[if lt IE 7]> <html class="ie6 lt-ie10 lt-ie9 lt-ie8 lt-ie7 no-js"
  lang="en-US"> <![endif]-->
< <!--[if IE 7]>    <html class="ie7 lt-ie10 lt-ie9 lt-ie8 no-js" lang="en-US">
  <![endif]-->
< <!--[if IE 8]>    <html class="ie8 lt-ie10 lt-ie9 no-js" lang="en-US">
  <![endif]-->
< <!--[if IE 9]>    <html class="ie9 lt-ie10 no-js" lang="en-US"> <![endif]-->
< <!--[if (gte IE 9)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js"
  lang="en-US" data-ntp-features="tracker-stats-widget:off"><!--<![endif]-->
---
> <!--[if IEMobile 7 ]> <html lang="en-GB" class="no-js iem7"> <![endif]-->
> <!--[if lt IE 7]> <html class="ie6 lt-ie10 lt-ie9 lt-ie8 lt-ie7 no-js"
  lang="en-GB"> <![endif]-->
> <!--[if IE 7]>    <html class="ie7 lt-ie10 lt-ie9 lt-ie8 no-js" lang="en-GB">
  <![endif]-->
> <!--[if IE 8]>    <html class="ie8 lt-ie10 lt-ie9 no-js" lang="en-GB">
  <![endif]-->
> <!--[if IE 9]>    <html class="ie9 lt-ie10 no-js" lang="en-GB"> <![endif]-->
> <!--[if (gte IE 9)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js"
  lang="en-GB" data-ntp-features="tracker-stats-widget:off"><!--<![endif]-->
48,49c48,49
<       <title>DuckDuckGo — Privacy, simplified.</title>
< <meta property="og:title" content="DuckDuckGo — Privacy, simplified." />
---
>       <title>DuckDuckGo â Privacy, simplified.</title>
> <meta property="og:title" content="DuckDuckGo â Privacy, simplified." />
64c64
< <script type="text/javascript" src="/locale/en_US/duckduckgo14.js"
  onerror="handleScriptError(this)"></script>
---
> <script type="text/javascript" src="/locale/en_GB/duckduckgo85.js"
  onerror="handleScriptError(this)"></script>
107c107
<                                               <!-- en_US All Settings -->
---
>                                               <!-- en_GB All Settings -->
146a147,148
> 
>
As you can see, the only real difference is the switch from en-US to en-GB, a one-character difference to the title of the page and the name of the locale file.

The result is also the same when viewing the page with either ESR 78 or the desktop browser: just a blank page.

Once again we find ourselves in an unsatisfactory position. But I will persevere and we will get to the bottom of this!

The next step is to check the network output from opening the page in the browser. And there's something important in there! There are many entries that look a bit like this: 
[ Request details ------------------------------------------- ]
    Request: GET status: 404 Not Found
    URL: https://www.flypig.co.uk/dist/o.2988a52fdfb14b7eff16.css
And now the penny drops: the page is expecting to be in the root of the domain. So while the location it's expecting to find the file is this: 
https://www.flypig.co.uk/dist/o.2988a52fdfb14b7eff16.css
I've instead been storing the file in this location: 
https://www.flypig.co.uk/tests/ddg8/dist/o.2988a52fdfb14b7eff16.css
Checking inside the index.html file the reason is clear. The paths are being given as absolute paths from the root of the domain, with a preceding slash, like this: 
<link rel="stylesheet" href="/dist/o.2988a52fdfb14b7eff16.css" type="text/css">
That / in front of the dist is causing all the trouble. It frustrates me that I didn't notice this before. But at least now I have something clear to fix. That'll be my task for tomorrow. Thankfully it should be really easy to fix. I feel a bit silly now.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
12 Jan 2024 : Day 136 #
Downloading all of the files served by DuckDuckGo individually didn't work: I end up with a site that simply triggers multiple "404 Not Found" errors due to files being requested but not available.

But I'm not giving up on this approach just yet. On the Sailfish Forum attah today made a nice suggestion in relation to this:
 
Finally remembering to post thoughts i have accumulated after the last weeks of following the blog: Have you tried with a wildly different User Agent override for DucDuckGo, like iPhone or something? The hanging parallel compile - could that be related to some syscall that gets used in synchronization, but which is stubbed in sb2?

There are actually two points here, the first or which relates to DuckDuckGo and the second of which relates to the issue of the build hanging when using more than one process using scratchbox2, part of the Sailfish SDK. Let me leave the compile query to one side for now, because, although it's a good point, I unfortunately don't know the answer (but it sounds like an interesting point to investigate).

Going back to DuckDuckGo, so far I've tried the ESR 78 user agent and the Firefox user agent, but I admit I've not tried anything else. It's a good idea — thank you attah — definitely worth trying. So let's see what happens.

I don't have an iPhone to compare with, but of course there are plenty of places on the Web that claim to list it. I'm going to use this one from the DeviceAtlas blog: 
Mozilla/5.0 (iPhone12,1; U; CPU iPhone OS 13_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/15E148 Safari/602.1
I used the Python script from yesterday to download the files twice, first using the ESR 78 user agent (stored to the ddg-esr78 directory) and then again using the iPhone user agent above (and stored to the ddg-iphone directory). Each directory contains 34 files and here's what I get when I diff them: 
$ find ddg-esr78/ | wc -l
34
$ find ddg-iphone12 | wc -l
34
$ diff --brief ddg-esr78/ ddg-iphone12/
Common subdirectories: ddg-esr78/assets and ddg-iphone12/assets
Common subdirectories: ddg-esr78/font and ddg-iphone12/font
Common subdirectories: ddg-esr78/ist and ddg-iphone12/ist
Common subdirectories: ddg-esr78/locale and ddg-iphone12/locale
So they resulting downloads are identical. That's too bad (although also a little reassuring). It's hard not to conclude that the user agent isn't the important factor here then. Nevertheless, I'm still concerned that I'm not getting the right files when I download using this Python script. If the problem is that DuckDuckGo is recognising a different browser when I download the files with my Python script — even if I've set the User Agent string to match — the the solution will have to be to download the files with the Sailfish Browser itself. It could be another issue entirely, but, well, this is a process of elimination.

I already have the means to do this, in theory. The EMBED_CONSOLE="network" setting gives a preview of any text files it downloads. But by default that's restricted to showing the first 32 KiB of data. That's not enough for everything and some files get truncated. So I've spent a bit of time improving the output.

First I've increased this value to 32 MiB. In practice I really want it to be set to have no limit, but 32 MiB should be more than enough (and if it isn't it should be obvious and can easily be bumped up). But when I first wrote this component I was always disappointed that the request and response headers could be output at a different time to the document content. That meant that it wasn't always possible to tie the content to the request headers (and in particular, the URL the content was downloaded from).

The reason the two can get separated is that the headers are output as soon as they've been received. And the content is output as soon as it's received in full. But between the headers being output and the content being received it's quite possible for some smaller file to be received in full. In this case, the smaller file would get printed out between the headers and content of the larger file.

My solution has been to store a copy of the URL in the content receiver callback object. That way, the URL can be output at the same time as the content. Now the headers and the content can be tied together since the URL is output with them both.

Here's an example (slightly abridged to keep things manageable): 
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/dist/p.f5b58579149e7488209f.js
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (Mobile; rv:78.0) Gecko/78.0 Firefox/78.0
        Accept : */*
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Referer : https://duckduckgo.com/
        Connection : keep-alive
        TE : Trailers
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Wed, 10 Jan 2024 22:27:17 GMT
        content-type : application/x-javascript
        content-length : 157
        last-modified : Fri, 27 Oct 2023 12:03:07 GMT
        vary : Accept-Encoding
        etag : "653ba6fb-9d"
        content-encoding : br
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : [...] ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Thu, 09 Jan 2025 22:27:17 GMT
        cache-control : max-age=31536000
        vary : Accept-Encoding
        X-Firefox-Spdy : h2
    [ Document URL ------------------------------------------ ]
        URL: https://duckduckgo.com/dist/p.f5b58579149e7488209f.js
        Charset: 
        ContentType: application/x-javascript
    [ Document content -------------------------------------- ]
function post(t){if(t.source===parent&&t.origin===location.protocol+"//"+
    location.hostname&&"string"==typeof t.data){var o=t.data.indexOf(":"),
    a=t.data.substr(0,o),n=t.data.substr(o+1);"ddg"===a&&(parent.window.
    location.href=n)}}window.addEventListener&&window.addEventListener
    ("message",post,!1);
    [ Document content ends --------------------------------- ]
Notice how the actual document content (which is only a few lines of text in this case) is right at the end. But directly beforehand the URL is output, which as a result can now be tied to the URL at the start of the request.

After downloading the mobile version of DuckDuckGo using the ESR 78 engine and these changes I can see they've made a difference when I compare the previous and newly collected data: 
$ ls -lh ddg-urls-esr78-mobile-*.txt
-rw-rw-r-- 1 flypig flypig 2.5K Jan  8 19:04 ddg-urls-esr78-mobile-01.txt
-rw-rw-r-- 1 flypig flypig 1.3M Jan 10 22:27 ddg-urls-esr78-mobile-02.txt
Previously 2.5 KiB of data was collected, but with these changes that goes up to 1.3 MiB.

The log file is a bit unwieldy, but it should hopefully contain all the data we need: every bit of textual data that was downloaded. Tomorrow I'll try to disentangle the output and turn them into files again. With a bit of luck, I'll end up with a working copy of the DuckDuckGo site (famous last words!).

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
11 Jan 2024 : Day 135 #
After collecting together and comparing the lists of files downloaded yesterday, today I'm actually downloading those files from the server.

I've created a very simply Python script that will take each line from the output, then reconstruct a local copy of each of the files, using the same relative directory hierarchy. The script is short and simple enough to show here in full. 
#!/bin/python3

import os
import urllib.request

BASE_DIR = 'ddg'
USER_AGENT = 'Mozilla/5.0 (Android 8.1.0; Mobile; rv:78.0) Gecko/78.0 Firefox/78.8'

def split_url(url):
	url = url.rstrip()
	path = url.lstrip('https://duckduckgo.com/')
	path, leaf = os.path.split(path)
	leaf = 'index.html' if not leaf else leaf
	path = os.path.join(BASE_DIR, path)
	filename = os.path.join(path, leaf)
	return path, filename, url

def make_dir(directory):
	print('Dir: {}'.format(directory))
	os.makedirs(directory, exist_ok=True)

def download_file(url, filename):
	print('URL: {}'.format(url))
	print('File: {}'.format(filename))
	opener = urllib.request.build_opener()
	opener.addheaders = [('User-agent', USER_AGENT)]
	urllib.request.install_opener(opener)
	urllib.request.urlretrieve(url, filename)

with open('download.txt') as fp:
	for line in fp:
		directory, filepath, url = split_url(line)
		make_dir(directory)
		download_file(url, filepath)
It really is a very linear process; they don't get much simpler than this. All it does is read in a file line by line. Each line is interpreted as a URL. For example suppose the line was the following: 
https://duckduckgo.com/dist/lib/l.656ceb337d61e6c36064.js
Then the file will extract the directory dist/lib/, create a local directory ddg/dist/lib/, then download the file from the URL and save it in the directory with the filename l.656ceb337d61e6c36064.js.

We'll end up with a directory structure that should match the root directory structure of DuckDuckGo: 
$ tree ddg
ddg
├── 3.html
├── assets
│   ├── logo_homepage.alt.v109.svg
│   ├── logo_homepage.normal.v109.svg
│   └── onboarding
│       ├── arrow.svg
│       └── bathroomguy
│           ├── 1-monster-v2--no-animation.svg
│           ├── 2-ghost-v2.svg
│           ├── 3-bathtub-v2--no-animation.svg
│           ├── 4-alpinist-v2.svg
│           └── teaser-2@2x.png
├── dist
│   ├── b.9e45618547aaad15b744.js
│   ├── d.01ff355796b8725c8dad.js
│   ├── h.2d6522d4f29f5b108aed.js
│   ├── lib
│   │   └── l.656ceb337d61e6c36064.js
│   ├── o.2988a52fdfb14b7eff16.css
│   ├── p.f5b58579149e7488209f.js
│   ├── s.b49dcfb5899df4f917ee.css
│   ├── ti.b07012e30f6971ff71d3.js
│   ├── tl.3db2557c9f124f3ebf92.js
│   └── util
│       └── u.a3c3a6d4d7bf9244744d.js
├── font
│   ├── ProximaNova-ExtraBold-webfont.woff2
│   ├── ProximaNova-Reg-webfont.woff2
│   └── ProximaNova-Sbold-webfont.woff2
├── index.html
└── locale
    └── en_GB
        └── duckduckgo85.js

9 directories, 24 files
The intention is that this will make a verbatim copy of all the files that the browser used when rendering the page. Unfortunately servers don't always serve the same file every time, but to try to avoid it serving up the wrong file, I've also set the user agent to be the same as for ESR 78.

That's no guarantee that the server will identify us as that — servers use all sorts of nasty tricks to try to identify misidentified browsers — but it's probably the best we can reasonably do.

Once I've got a local copy of the site structure I copy this over to my server and get the browser to render it.

But unfortunately without success. For reasons I can't figure out, when I attempt to open the page, the browser requests a wholly different set of files to download. And not just different leafnames, but a totally different file structure as well. So rather than it downloading the files I've collected together, I just get a bunch of "404 File Not Found" errors.

Frustrating. But the nature of me writing this up daily is that I can't just summarise all the things that work. As anyone who's been following along will no doubt have noticed by now, often things I try just don't work. But from the comments I've been getting from others, it's reassuring to know it's not just me. Sometimes failure is still progress.

Maybe I'll have better luck with a new approach tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
10 Jan 2024 : Day 134 #
I was getting a little despondent trying to fix DuckDuckGo on ESR 91, but the tests I performed yesterday have invigorated me somewhat. It's now clear that there's a decent quantity of accesses being made when using ESR 91 and that wouldn't be happening unless at least some of the data was being interpreted as HTML.

But it's also clear that not everything is as it should be: on ESR 78 there are a mixture of SVG and PNG files being downloaded to provide the images on the page. In contrast, on ESR 91, there are no images being downloaded at all. There are two possible reasons for this I can think of:
  1. DuckDuckGo is serving pages that don't contain any images. Seems unlikely, but nevertheless a possibility.
  2. There are images but ESR 91 isn't turning them in to access requests. I have no idea why this might happen, yet it still feels the more likely of the two scenarios.
I think it would be useful to know how much overlap there is between the sets of files that are being downloaded. So I put together a command line monstrosity to compare them. Before giving the final command, let me break down what it's doing.

First, there are some extraneous accesses in the list that have nothing to do with DuckDuckGo but are a consequence of gecko collecting settings data after the profile was wiped; lines like this: 
https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/
    changes/records?collection=hijack-blocklists&bucket=main
So the command filters all of the lines that don't include duckduckgo.

Second, I noticed that some files appear to be the same but located at different URLs. For example, this can be found in the ERS 78 list: 
https://duckduckgo.com/font/ProximaNova-ExtraBold-webfont.woff2
While this can be found in the ESR 91 list: 
https://duckduckgo.com/static-assets/font/ProximaNova-Reg-webfont.woff2
These are both font files; they must surely be the same file, right? But they're located in different folders. So the command then strips the URLs down to the leafnames.

Third, it then sorts the results alphabetically so that any identical lines in both files will appear in the same order. If there are any matching lines, this will make any diff of the two much cleaner.

Finally the command then performs a side-by-side diff on the result. Here's all of that put together: 
$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" log1.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" log2.txt) | sort)
When I came up with this approach I thought it would give amazing results, but in practice it's not as exciting as I was hoping for.

Let's concentrate on the mobile version of the page first. Here's the diff: 
$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr78-mobile.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr91-mobile.txt) | sort)

1-monster-v2--no-animation.svg             | 18040-1287342b1f839f70.js
2-ghost-v2.svg                             | 38407-070351ade350c8e4.js
3-bathtub-v2--no-animation.svg             | 39337-cd8caeeff0afb1c4.js
4-alpinist-v2.svg                          | 41966-c9d76895b4f9358f.js
arrow.svg                                  | 55015-29fec414530c2cf6.js
b.9e45618547aaad15b744.js                  | 55672-19856920a309aea5.js
d.01ff355796b8725c8dad.js                  | 61754-29df12bb83d71c7b.js
duckduckgo85.js                            | 6a4833195509cc3d.css
h.2d6522d4f29f5b108aed.js                  | 703c9a9a057785a9.css
hi?7857271&b=firefox&ei=true&i=false&[...] | 81125-b74d1b6f4908497b.js
l.656ceb337d61e6c36064.js                  | 93432-ebd443fe69061b19.js
logo_homepage.alt.v109.svg                 | 94623-d5bfa67fc3bada59.js
logo_homepage.normal.v109.svg              | 95665-30dd494bea911abd.js
o.2988a52fdfb14b7eff16.css                 | a2a29f84956f2aac.css
p.f5b58579149e7488209f.js                  | _app-ce0b94ea69138577.js
post3.html                                 | _buildManifest.js
                                           > c89114cfe55133c4.css
                                           > ed8494aa71104fdc.css
                                           > f0b3f7da285c9dbd.css
                                           > framework-f8115f7fae64930e.js
                                           > home-34dda07336cb6ee1.js
                                           > main-17a05b704438cdd6.js
                                           > ProximaNova-Bold-webfont.woff2
ProximaNova-ExtraBold-webfont.woff2          ProximaNova-ExtraBold-webfont.woff2
                                           > ProximaNova-RegIt-webfont.woff2
ProximaNova-Reg-webfont.woff2                ProximaNova-Reg-webfont.woff2
ProximaNova-Sbold-webfont.woff2              ProximaNova-Sbold-webfont.woff2
s.b49dcfb5899df4f917ee.css                 | _ssgManifest.js
teaser-2@2x.png                            | webpack-7358ea7cdec0aecf.js
ti.b07012e30f6971ff71d3.js                 <
tl.3db2557c9f124f3ebf92.js                 <
u.a3c3a6d4d7bf9244744d.js                  <
Only three files are shared across the two collections. The remaining 22 and 27 files respectively are apparently different. I was honestly hoping for there to be more similarity.

For completeness let's do the same for the desktop collections: 
$ diff --side-by-side \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr78-desktop.txt) | sort) \
    <(sed -e 's/^.*\/\(.*\)/\1/g' <(grep "duckduckgo" ddg-urls-esr91-d.txt) | sort)

18040-1287342b1f839f70.js                    18040-1287342b1f839f70.js
38407-070351ade350c8e4.js                    38407-070351ade350c8e4.js
39337-cd8caeeff0afb1c4.js                    39337-cd8caeeff0afb1c4.js
41966-c9d76895b4f9358f.js                    41966-c9d76895b4f9358f.js
48292.8c8d6cb394d25a15.js                  <
55015-29fec414530c2cf6.js                    55015-29fec414530c2cf6.js
55672-19856920a309aea5.js                    55672-19856920a309aea5.js
61754-29df12bb83d71c7b.js                    61754-29df12bb83d71c7b.js
6a4833195509cc3d.css                         6a4833195509cc3d.css
703c9a9a057785a9.css                         703c9a9a057785a9.css
81125-b74d1b6f4908497b.js                    81125-b74d1b6f4908497b.js
93432-ebd443fe69061b19.js                    93432-ebd443fe69061b19.js
94623-d5bfa67fc3bada59.js                    94623-d5bfa67fc3bada59.js
95665-30dd494bea911abd.js                    95665-30dd494bea911abd.js
a2a29f84956f2aac.css                         a2a29f84956f2aac.css
add-firefox.f0890a6c.svg                   <
_app-ce0b94ea69138577.js                     _app-ce0b94ea69138577.js
app-protection-back-dark.png               <
app-protection-front-dark.png              <
app-protection-ios-dark.png                <
app-store.501fe17a.png                     <
atb_home_impression?9836955&b=firefox[...] <
_buildManifest.js                            _buildManifest.js
burn@2x.be0bd36d.png                       <
c89114cfe55133c4.css                         c89114cfe55133c4.css
chrome-lg.a4859fb2.png                     <
CNET-DARK.e3fd496e.png                     <
dark-mode@2x.3e150d01.png                  <
devices-dark.png                           <
ed8494aa71104fdc.css                         ed8494aa71104fdc.css
edge-lg.36af7682.png                       <
email-protection-back-dark.png             <
email-protection-front-light.png           <
email-protection-ios-dark.png              <
f0b3f7da285c9dbd.css                         f0b3f7da285c9dbd.css
firefox-lg.8efad702.png                    <
flame.1241f020.png                         <
flame@2x.40e1cfa0.png                      <
flame-narrow.70589b7c.png                  <
framework-f8115f7fae64930e.js                framework-f8115f7fae64930e.js
home-34dda07336cb6ee1.js                     home-34dda07336cb6ee1.js
legacy-homepage-btf-dark.png               <
legacy-homepage-btf-mobile-dark.png        <
macos.61889438.png                         <
main-17a05b704438cdd6.js                     main-17a05b704438cdd6.js
night@2x.4ca79636.png                      <
opera-lg.237c4418.png                      <
page_home_commonImpression?2448534&[...]   <
play-store.e5d5ed36.png                    <
ProximaNova-Bold-webfont.woff2               ProximaNova-Bold-webfont.woff2
ProximaNova-ExtraBold-webfont.woff2          ProximaNova-ExtraBold-webfont.woff2
ProximaNova-RegIt-webfont.woff2              ProximaNova-RegIt-webfont.woff2
ProximaNova-Reg-webfont.woff2                ProximaNova-Reg-webfont.woff2
ProximaNova-Sbold-webfont.woff2              ProximaNova-Sbold-webfont.woff2
safari-lg.8406694a.png                     <
search-protection-back-light.png           <
search-protection-front-dark.png           <
search-protection-ios-dark.png             <
set-as-default.d95c3465.svg                <
_ssgManifest.js                              _ssgManifest.js
UT-DARK-DEFAULT.6cd0020d.png               <
VERGE-DARK-DEFAULT.8850a2d2.png            <
webpack-7358ea7cdec0aecf.js                  webpack-7358ea7cdec0aecf.js
web-protection-back-dark.png               <
web-protection-front-dark.png              <
web-protection-ios-dark.png                <
widget-big@2x.a260ccf6.png                 <
widget-small@2x.07c865df.png               <
windows.477fa143.png                       <
WIRED-DARK-DEFAULT.b4d48a49.png            <
Here we see something more interesting. There are 30 files the same across the two collections with 43 and 0 files being different respectively. In other words, the ESR 91 collection is a subset of the ESR 78 collection.

There might be something in this, but initially I'm more interested in the mobile version of the site and there the overlap is far less. However, now that that I have the URLs, one thing I can do is download all of the files and try to use them to recreate the site on my own server. It's possible this will give better results than saving out the files from the desktop browser, so I'll be giving that a go tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
9 Jan 2024 : Day 133 #
It's time to pick up from where we left off yesterday, hanging on a couple of breakpoints on two different phones, one running ESR 78 and the other running ESR 91.

It feels weird leaving the phones in this state of limbo overnight. Astonishingly, even though I put my laptop to sleep, the SSH connections weren't dropped. Returning to discover things in exactly the same state I left them in is disconcerting. But in a good way.

Before I get on to the next steps I also want to take the time to acknowledge the help of TrickyPR who sent this helpful message via Mastodon:
 
my knowledge of the fancy debugger is a lot more modern, but here are some tips. For the server:
  • make sure you are building devtools/ (you probably already are)
  • you can launch the devtools server through js: https://github.com/ajvincent/motherhen/blob/main/cleanroom/source/modules/DevtoolsServer.jsm
  • you will have better luck with about:debugging on a similar version browser
client (browser toolbox):
  • you need to set MOZ_DEVTOOLS to all
  • you will need to implement a command line handler for -chrome (you can steal this, but it’s esm (needs esr +100?): https://github.com/ajvincent/motherhen/pull/34 ) Feel free to ping if you want clarification or get stuck.

This is really helpful. I've not had a chance to look into the remote debugging (I think it'll be something for the weekend) so please bear with me, but it looks like this info will definitely make for reference material.

To recap, yesterday we were stepping through nsDisplayBackgroundImage::AppendBackgroundItemsToTop() because we know that on ESR 78 we eventually reach a call to nsDisplayBackgroundImage::GetInitData(), whereas on ESR 91 this method is never called. The two versions of AppendBackgroundItemsToTop() have changed between ESR 78 and ESR 91, but they're close enough to be able to follow them both side-by-side.

But stepping through didn't work as well as I'd hoped because for the first time the breakpoint is hit on both devices, the method returns early in both cases. The AppendBackgroundItemsToTop() method gets called multiple times during a render, so it's not surprising that in some cases it returns early and in others it goes all the way through to GetInitData(). I need a call on ESR 78 that goes all the way to GetInitData() and — and here's the tricky bit — I need the same call on ESR 91 that returns early.

To get closer to this situation I'm going to move the breakpoints closer to the call to GetInitData() on both systems.

Here's the relevant bit of code, which is pretty similar on both ESR 78 and ESR 91, but this copy happens to come from ESR 91: 
  if (!bg || !drawBackgroundImage) {
    if (!bgItemList.IsEmpty()) {
      aList->AppendToTop(&bgItemList);
      return AppendedBackgroundType::Background;
    }

    return AppendedBackgroundType::None;
  }

  const ActiveScrolledRoot* asr = aBuilder->CurrentActiveScrolledRoot();

  bool needBlendContainer = false;

  // Passing bg == nullptr in this macro will result in one iteration with
  // i = 0.
  NS_FOR_VISIBLE_IMAGE_LAYERS_BACK_TO_FRONT(i, bg->mImage) {
    if (bg->mImage.mLayers[i].mImage.IsNone()) {
      continue;
    }
[...]

    nsDisplayList thisItemList;
    nsDisplayBackgroundImage::InitData bgData =
        nsDisplayBackgroundImage::GetInitData(aBuilder, aFrame, i, bgOriginRect,
                                              bgSC);
I've chopped a bit out, but there's no way for the method to return in the part I've removed, so the bits shown here are the important pieces for our purposes today.

The following line is the last statement before the loop around the image layers is entered. 
  bool needBlendContainer = false;
If I place a breakpoint on this line, on ESR 91 the breakpoint remains untouched: 
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep n   0x0000007fbc3511d4 in nsDisplayBackgroundImage::
        GetInitData(nsDisplayListBuilder*, nsIFrame*, unsigned short,
        nsRect const&, mozilla::ComputedStyle*)
        at layout/painting/nsDisplayList.cpp:3409
2       breakpoint     keep y   0x0000007fbc3a8730 in nsDisplayBackgroundImage::
        AppendBackgroundItemsToTop(nsDisplayListBuilder*, nsIFrame*,
        nsRect const&, nsDisplayList*, bool, mozilla::ComputedStyle*,
        nsRect const&, nsIFrame*,
        mozilla::Maybe<nsDisplayListBuilder::AutoBuildingDisplayList>*) 
        at layout/painting/nsDisplayList.cpp:3632
        breakpoint already hit 2 times
(gdb) handle SIGPIPE nostop
Signal        Stop      Print   Pass to program Description
SIGPIPE       No        Yes     Yes             Broken pipe
(gdb) disable break 2
(gdb) break nsDisplayList.cpp:3766
Breakpoint 3 at 0x7fbc3a9344: file layout/painting/nsDisplayList.cpp, line 3766.
(gdb) c
[...]
So the method must be being exited — in all cases — before this line occurs. From my recollection of stepping through the method earlier I'm pretty sure we reached this condition: 
  if (!bg || !drawBackgroundImage) {
So I'll put a breakpoint there and try again. 
(gdb) disable break 3
(gdb) break nsDisplayList.cpp:3755
Breakpoint 4 at 0x7fbc3a8dac: file layout/painting/nsDisplayList.cpp, line 3755.
(gdb) c
Continuing.
[New LWP 28203]
[LWP 27764 exited]
[LWP 28203 exited]
[Switching to LWP 31016]

Thread 10 "GeckoWorkerThre" hit Breakpoint 4, nsDisplayBackgroundImage::
    AppendBackgroundItemsToTop (aBuilder=0x7f9efa2378, aFrame=0x7f803c9eb8, 
    aBackgroundRect=..., aList=0x7f9ef9ff08, aAllowWillPaintBorderOptimization=
    <optimized out>, aComputedStyle=<optimized out>, aBackgroundOriginRect=...,
    aSecondaryReferenceFrame=0x0, aAutoBuildingDisplayList=0x0)
    at layout/painting/nsDisplayList.cpp:3755
3755      if (!bg || !drawBackgroundImage) {
(gdb)
Now it hits. Let's try putting some breakpoints on the return calls. 
(gdb) disable break 4
(gdb) break nsDisplayList.cpp:3758
Note: breakpoint 3 (disabled) also set at pc 0x7fbc3a9344.
Breakpoint 5 at 0x7fbc3a9344: file layout/painting/nsDisplayList.cpp, line 3766.
(gdb) break nsDisplayList.cpp:3761
Note: breakpoints 3 (disabled) and 5 also set at pc 0x7fbc3a9344.
Breakpoint 6 at 0x7fbc3a9344: file layout/painting/nsDisplayList.cpp, line 3766.
(gdb)
That's not helpful: the debugger won't let me place a breakpoint on either of these lines, presumably because after compilation to machine code it's no longer possible to distinguish between the lines properly.

However, if I leave the breakpoint on the start of the condition I notice that during rendering the breakpoint is hit precisely four times on ESR 91.

In contrast, when rendering the page on ESR 78 the breakpoint is hit 108 times. That's a red flag, because it really suggests that there are a lot of background images being rendered on ESR 78, and almost no attempt to even render backgrounds on ESR 91.

What's more, on ESR 91, even though we can't place a breakpoint on the return lines, we can infer the value that's being returned by looking at the values of the variables going in to the condition. Here's the condition again on ESR 91 for reference: 
  if (!bg || !drawBackgroundImage) {
    if (!bgItemList.IsEmpty()) {
      aList->AppendToTop(&bgItemList);
      return AppendedBackgroundType::Background;
    }

    return AppendedBackgroundType::None;
  }
And here are the values the debugger will give us access to: 
Thread 10 "GeckoWorkerThre" hit Breakpoint 9, nsDisplayBackgroundImage::
    AppendBackgroundItemsToTop (aBuilder=0x7f9efa2378, aFrame=0x7f80b5fa48, 
    aBackgroundRect=..., aList=0x7f9ef9ff08, aAllowWillPaintBorderOptimization=
    <optimized out>, aComputedStyle=<optimized out>, aBackgroundOriginRect=...,
    aSecondaryReferenceFrame=0x0, aAutoBuildingDisplayList=0x0)
    at layout/painting/nsDisplayList.cpp:3755
3755      if (!bg || !drawBackgroundImage) {
(gdb) p bg
$8 = <optimized out>
(gdb) p drawBackgroundImage
$9 = false
(gdb) p bgItemList.mLength
$19 = 0
(gdb) c
Continuing.
I get the same results for all four cases and working through the logic, if drawBackgroundImage is set to false then !drawBackgroundImage will be set to true, hence (!bg || !drawBackgroundImage) will be set to true, hence the condition will be entered into. At that point we know that bgItemList has no members and so will be empty, so the nested condition won't be entered.

The method will then return with a value of AppendedBackgroundType::None.

In contrast we know that on ESR 78 execution gets beyond this condition in order to actually render background images, just as we would expect for any relatively complex page.

Given all of the above it looks to me very much like the problem isn't happening in the render loop. Rendering is happening, there just isn't anything to render. There could be multiple reasons for this. One possibility is that the page is being received but somehow lost or dropped. The other is that the page isn't being sent in full. It could be JavaScript related.

To return to something discussed earlier, this behaviour distinguishes the problem from the issue we experience with Amazon on ESR 78. In that case, rendering most definitely occurs because we briefly see a copy of the page. The page visibly disappears in front of our eyes. In that case we'd expect there to be many breakpoint hits with rendering of multiple background images all prior to the page disappearing.

To make more sense of what's going on I need to go back down to the other end of the stack: the networking end. I realised after some thought that the experiments I ran earlier with EMBED_CONSOLE="network" set were not very effective. What I really want is a list of all the network access requests made from the browser in order to understand where the problem is happening. For example, are there any images being downloaded?

The last time I did this there weren't any images downloaded, but in retrospect that might have been because they were being cached. I should do this more robustly, and because I'm really only interested in the URLs, I can filter out all of the other details to make my life easier.

There are two reasons for doing this. First to compare with the URLs accessed using ESR 78, which is something I also failed to do previously. Second because if I have all the URLs, that might also help me piece together a full version of the page without all of the mysterious transformations that happen when I save the page manually from the desktop browser.

So to kick things off, here are all of the URLs accessed for ESR 78 when requesting the mobile version of the site. Note that the first thing I do is delete the profile entirely. That's to avoid anything being cached.

Mobile version of the site on ESR 78 
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser https://duckduckgo.com/ 2>&1 | grep "URL: "
URL: https://duckduckgo.com/
URL: https://duckduckgo.com/dist/s.b49dcfb5899df4f917ee.css
URL: https://duckduckgo.com/dist/o.2988a52fdfb14b7eff16.css
URL: https://duckduckgo.com/dist/tl.3db2557c9f124f3ebf92.js
URL: https://duckduckgo.com/dist/b.9e45618547aaad15b744.js
URL: https://duckduckgo.com/dist/lib/l.656ceb337d61e6c36064.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/records?collection=hijack-blocklists&bucket=main
URL: https://duckduckgo.com/locale/en_GB/duckduckgo85.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/records?collection=anti-tracking-url-decoration&bucket=main
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/records?collection=url-classifier-skip-urls&bucket=main
URL: https://duckduckgo.com/dist/util/u.a3c3a6d4d7bf9244744d.js
URL: https://duckduckgo.com/dist/d.01ff355796b8725c8dad.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/
     collections/hijack-blocklists/changeset?_expected=1605801189258
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/
     collections/url-classifier-skip-urls/changeset?_expected=1701090424142
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/
     collections/anti-tracking-url-decoration/changeset?_expected=1564511755134
URL: https://duckduckgo.com/dist/h.2d6522d4f29f5b108aed.js
URL: https://duckduckgo.com/dist/ti.b07012e30f6971ff71d3.js
URL: https://duckduckgo.com/font/ProximaNova-Reg-webfont.woff2
URL: https://content-signature-2.cdn.mozilla.net/chains/
     remote-settings.content-signature.mozilla.org-2024-02-08-20-06-04.chain
URL: https://duckduckgo.com/post3.html
URL: https://duckduckgo.com/assets/logo_homepage.normal.v109.svg
URL: https://duckduckgo.com/font/ProximaNova-Sbold-webfont.woff2
URL: https://duckduckgo.com/assets/onboarding/bathroomguy/teaser-2@2x.png
URL: https://duckduckgo.com/assets/onboarding/arrow.svg
URL: https://duckduckgo.com/assets/logo_homepage.alt.v109.svg
URL: https://duckduckgo.com/font/ProximaNova-ExtraBold-webfont.woff2
URL: https://duckduckgo.com/assets/onboarding/bathroomguy/
     1-monster-v2--no-animation.svg
URL: https://duckduckgo.com/assets/onboarding/bathroomguy/2-ghost-v2.svg
URL: https://duckduckgo.com/assets/onboarding/bathroomguy/
     3-bathtub-v2--no-animation.svg
URL: https://duckduckgo.com/assets/onboarding/bathroomguy/4-alpinist-v2.svg
URL: https://duckduckgo.com/dist/p.f5b58579149e7488209f.js
URL: https://improving.duckduckgo.com/t/hi?7857271&b=firefox&ei=true&i=false&
     d=m&l=en-GB&p=other&pre_atb=v411-5&ax=false&ak=false&pre_va=_&pre_atbva=_
Now that's a lot more accesses than I had last time, which is a good sign. That's 32 URL accesses in total. Interestingly all bar one of the images are SVG format: all vector apart from a single bitmap.

It's also worth noting that some of the requests are to Mozilla servers rather than DuckDuckGo servers. I suspect that's a consequence of having wiped the profile: the browser is making accesses to retrieve some of the data that I deleted. We should ignore those requests.

Next up the desktop version of the same site. These were generated by selecting the "Desktop Mode" button in the browser immediately after downloading the mobile site. Consequently it's possible some data was cached.

Desktop version of the site on ESR 78 
URL: https://duckduckgo.com/
URL: https://duckduckgo.com/_next/static/css/c89114cfe55133c4.css
URL: https://duckduckgo.com/_next/static/css/6a4833195509cc3d.css
URL: https://duckduckgo.com/_next/static/css/a2a29f84956f2aac.css
URL: https://duckduckgo.com/_next/static/css/f0b3f7da285c9dbd.css
URL: https://duckduckgo.com/_next/static/css/ed8494aa71104fdc.css
URL: https://duckduckgo.com/_next/static/css/703c9a9a057785a9.css
URL: https://duckduckgo.com/_next/static/chunks/webpack-7358ea7cdec0aecf.js
URL: https://duckduckgo.com/_next/static/chunks/framework-f8115f7fae64930e.js
URL: https://duckduckgo.com/_next/static/chunks/main-17a05b704438cdd6.js
URL: https://duckduckgo.com/_next/static/chunks/pages/_app-ce0b94ea69138577.js
URL: https://duckduckgo.com/_next/static/chunks/41966-c9d76895b4f9358f.js
URL: https://duckduckgo.com/_next/static/chunks/93432-ebd443fe69061b19.js
URL: https://duckduckgo.com/_next/static/chunks/18040-1287342b1f839f70.js
URL: https://duckduckgo.com/_next/static/chunks/81125-b74d1b6f4908497b.js
URL: https://duckduckgo.com/_next/static/chunks/39337-cd8caeeff0afb1c4.js
URL: https://duckduckgo.com/_next/static/chunks/94623-d5bfa67fc3bada59.js
URL: https://duckduckgo.com/_next/static/chunks/95665-30dd494bea911abd.js
URL: https://duckduckgo.com/_next/static/chunks/55015-29fec414530c2cf6.js
URL: https://duckduckgo.com/_next/static/chunks/61754-29df12bb83d71c7b.js
URL: https://duckduckgo.com/_next/static/chunks/55672-19856920a309aea5.js
URL: https://duckduckgo.com/_next/static/chunks/38407-070351ade350c8e4.js
URL: https://duckduckgo.com/_next/static/chunks/pages/%5Blocale%5D/
     home-34dda07336cb6ee1.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_buildManifest.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_ssgManifest.js
URL: https://improving.duckduckgo.com/t/page_home_commonImpression?2448534&
     b=firefox&d=d&l=en-GB&p=linux&atb=v411-5&pre_va=_&pre_atbva=_&atbi=true&
     i=false&ak=false&ax=false
URL: https://duckduckgo.com/_next/static/media/set-as-default.d95c3465.svg
URL: https://duckduckgo.com/static-assets/image/pages/legacy-home/
     devices-dark.png
URL: https://duckduckgo.com/static-assets/backgrounds/
     legacy-homepage-btf-mobile-dark.png
URL: https://duckduckgo.com/_next/static/media/macos.61889438.png
URL: https://duckduckgo.com/_next/static/media/windows.477fa143.png
URL: https://duckduckgo.com/_next/static/media/app-store.501fe17a.png
URL: https://duckduckgo.com/_next/static/media/play-store.e5d5ed36.png
URL: https://duckduckgo.com/_next/static/media/chrome-lg.a4859fb2.png
URL: https://duckduckgo.com/_next/static/media/edge-lg.36af7682.png
URL: https://duckduckgo.com/_next/static/media/firefox-lg.8efad702.png
URL: https://duckduckgo.com/_next/static/media/opera-lg.237c4418.png
URL: https://duckduckgo.com/_next/static/media/safari-lg.8406694a.png
URL: https://duckduckgo.com/_next/static/chunks/48292.8c8d6cb394d25a15.js
URL: https://duckduckgo.com/static-assets/backgrounds/legacy-homepage-btf-dark.png
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Reg-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-ExtraBold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Bold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Sbold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-RegIt-webfont.woff2
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/search-protection-back-light.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/search-protection-front-dark.png
URL: https://duckduckgo.com/_next/static/media/flame.1241f020.png
URL: https://duckduckgo.com/_next/static/media/burn@2x.be0bd36d.png
URL: https://duckduckgo.com/_next/static/media/flame@2x.40e1cfa0.png
URL: https://duckduckgo.com/_next/static/media/widget-big@2x.a260ccf6.png
URL: https://duckduckgo.com/_next/static/media/night@2x.4ca79636.png
URL: https://duckduckgo.com/_next/static/media/dark-mode@2x.3e150d01.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/email-protection-front-light.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/app-protection-back-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/app-protection-front-dark.png
URL: https://duckduckgo.com/_next/static/media/flame-narrow.70589b7c.png
URL: https://duckduckgo.com/_next/static/media/widget-small@2x.07c865df.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     search-protection-ios-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     web-protection-ios-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     email-protection-ios-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/web-protection-back-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/web-protection-front-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     desktop/email-protection-back-dark.png
URL: https://duckduckgo.com/static-assets/image/pages/home/devices/how-it-works/
     app-protection-ios-dark.png
URL: https://duckduckgo.com/_next/static/media/add-firefox.f0890a6c.svg
URL: https://duckduckgo.com/_next/static/media/WIRED-DARK-DEFAULT.b4d48a49.png
URL: https://duckduckgo.com/_next/static/media/VERGE-DARK-DEFAULT.8850a2d2.png
URL: https://duckduckgo.com/_next/static/media/UT-DARK-DEFAULT.6cd0020d.png
URL: https://duckduckgo.com/_next/static/media/CNET-DARK.e3fd496e.png
URL: https://improving.duckduckgo.com/t/atb_home_impression?9836955&b=firefox&
     d=d&l=en-GB&p=linux&atb=v411-5&pre_va=_&pre_atbva=_&atbi=true&i=false&
     ak=false&ax=false
The desktop version generates many more access requests, 71 in total, including a large number of PNG files (I count 36 in total). These are the results for ESR 78, so that's a working copy of the site. On the face of it there shouldn't be any reason for ESR 91 to be served anything different to this.

So now using ESR 91, here are the URLs accessed with an empty profile and the mobile version of the site.

Mobile version of the site on ESR 91 
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser https://duckduckgo.com/ 2>&1 | grep "URL: "
URL: http://detectportal.firefox.com/success.txt?ipv4
URL: https://duckduckgo.com/
URL: https://location.services.mozilla.com/v1/country?key=no-mozilla-api-key
URL: https://duckduckgo.com/static-assets/font/ProximaNova-RegIt-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-ExtraBold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Reg-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Sbold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Bold-webfont.woff2
URL: https://duckduckgo.com/_next/static/css/c89114cfe55133c4.css
URL: https://duckduckgo.com/_next/static/css/6a4833195509cc3d.css
URL: https://duckduckgo.com/_next/static/css/a2a29f84956f2aac.css
URL: https://duckduckgo.com/_next/static/css/f0b3f7da285c9dbd.css
URL: https://duckduckgo.com/_next/static/css/ed8494aa71104fdc.css
URL: https://duckduckgo.com/_next/static/css/703c9a9a057785a9.css
URL: https://duckduckgo.com/_next/static/chunks/webpack-7358ea7cdec0aecf.js
URL: https://duckduckgo.com/_next/static/chunks/framework-f8115f7fae64930e.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=anti-tracking-url-decoration&
     bucket=main&_expected=0
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=query-stripping&bucket=main&
     _expected=0
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=hijack-blocklists&
     bucket=main&_expected=0
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=url-classifier-skip-urls&
     bucket=main&_expected=0
URL: https://duckduckgo.com/_next/static/chunks/main-17a05b704438cdd6.js
URL: https://duckduckgo.com/_next/static/chunks/pages/_app-ce0b94ea69138577.js
URL: https://duckduckgo.com/_next/static/chunks/41966-c9d76895b4f9358f.js
URL: https://duckduckgo.com/_next/static/chunks/93432-ebd443fe69061b19.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=password-recipes&bucket=main&
     _expected=0
URL: https://duckduckgo.com/_next/static/chunks/18040-1287342b1f839f70.js
URL: https://duckduckgo.com/_next/static/chunks/81125-b74d1b6f4908497b.js
URL: https://duckduckgo.com/_next/static/chunks/39337-cd8caeeff0afb1c4.js
URL: https://duckduckgo.com/_next/static/chunks/94623-d5bfa67fc3bada59.js
URL: https://duckduckgo.com/_next/static/chunks/95665-30dd494bea911abd.js
URL: https://duckduckgo.com/_next/static/chunks/55015-29fec414530c2cf6.js
URL: https://duckduckgo.com/_next/static/chunks/61754-29df12bb83d71c7b.js
URL: https://duckduckgo.com/_next/static/chunks/55672-19856920a309aea5.js
URL: https://duckduckgo.com/_next/static/chunks/38407-070351ade350c8e4.js
URL: https://duckduckgo.com/_next/static/chunks/pages/%5Blocale%5D/
     home-34dda07336cb6ee1.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_buildManifest.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_ssgManifest.js
URL: https://firefox.settings.services.mozilla.com/v1/buckets/monitor/
     collections/changes/changeset?collection=search-config&bucket=main&
     _expected=0
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     anti-tracking-url-decoration/changeset?_expected=1564511755134
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     query-stripping/changeset?_expected=1694689843914
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     hijack-blocklists/changeset?_expected=1605801189258
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     password-recipes/changeset?_expected=1674595048726
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     url-classifier-skip-urls/changeset?_expected=1701090424142
URL: https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/
     search-config/changeset?_expected=1701806851414
URL: https://content-signature-2.cdn.mozilla.net/chains/
     remote-settings.content-signature.mozilla.org-2024-02-08-20-06-04.chain
Now that's certainly more files than I thought previously were being accessed using ESR 91. In fact its 45 access requests, which is more than we saw for ESR 78. What about the desktop version.

Desktop version of the site on ESR 91 
URL: https://duckduckgo.com/
URL: https://duckduckgo.com/static-assets/font/ProximaNova-RegIt-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-ExtraBold-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Reg-webfont.woff2
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Sbold-webfont.woff2
URL: https://duckduckgo.com/_next/static/css/c89114cfe55133c4.css
URL: https://duckduckgo.com/_next/static/css/6a4833195509cc3d.css
URL: https://duckduckgo.com/_next/static/css/a2a29f84956f2aac.css
URL: https://duckduckgo.com/_next/static/css/f0b3f7da285c9dbd.css
URL: https://duckduckgo.com/_next/static/css/ed8494aa71104fdc.css
URL: https://duckduckgo.com/_next/static/css/703c9a9a057785a9.css
URL: https://duckduckgo.com/static-assets/font/ProximaNova-Bold-webfont.woff2
URL: https://duckduckgo.com/_next/static/chunks/webpack-7358ea7cdec0aecf.js
URL: https://duckduckgo.com/_next/static/chunks/framework-f8115f7fae64930e.js
URL: https://duckduckgo.com/_next/static/chunks/main-17a05b704438cdd6.js
URL: https://duckduckgo.com/_next/static/chunks/pages/_app-ce0b94ea69138577.js
URL: https://duckduckgo.com/_next/static/chunks/41966-c9d76895b4f9358f.js
URL: https://duckduckgo.com/_next/static/chunks/93432-ebd443fe69061b19.js
URL: https://duckduckgo.com/_next/static/chunks/18040-1287342b1f839f70.js
URL: https://duckduckgo.com/_next/static/chunks/81125-b74d1b6f4908497b.js
URL: https://duckduckgo.com/_next/static/chunks/39337-cd8caeeff0afb1c4.js
URL: https://duckduckgo.com/_next/static/chunks/94623-d5bfa67fc3bada59.js
URL: https://duckduckgo.com/_next/static/chunks/95665-30dd494bea911abd.js
URL: https://duckduckgo.com/_next/static/chunks/55015-29fec414530c2cf6.js
URL: https://duckduckgo.com/_next/static/chunks/61754-29df12bb83d71c7b.js
URL: https://duckduckgo.com/_next/static/chunks/55672-19856920a309aea5.js
URL: https://duckduckgo.com/_next/static/chunks/38407-070351ade350c8e4.js
URL: https://duckduckgo.com/_next/static/chunks/pages/%5Blocale%5D/
     home-34dda07336cb6ee1.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_buildManifest.js
URL: https://duckduckgo.com/_next/static/ZNJGkb3SCV-nLlzz3kvNx/_ssgManifest.js
Only 30 accesses. That's really unexpected. I'm also going to collect myself a version of the full log output in case I need to refer back to it. First on ESR 78: 
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser https://duckduckgo.com/ \
  2>&1 > log-ddg-78.txt
Then also on ESR 91: 
$ rm -rf ~/.local/share/org.sailfishos/browser
$ EMBED_CONSOLE="network" sailfish-browser https://duckduckgo.com/ \
  2>&1 > log-ddg-91.txt
The next step will be to compare the results from ESR 78 with those from ESR 91 properly. That's where I'll pick this up tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
8 Jan 2024 : Day 132 #
For the last twenty hours or so my development phone has been loyally running ESR 91 in the debugger. That's because I've been searching for a suitable breakpoint to distinguish a working render of DuckDuckGo from a failing render of DuckDuckGo.

The time spent in between has been fruitful, but before I get on to that I want to first thank simonschmeisser for the useful comment on the Sailfish Forum about remote debugging. Simon highlights the fact that Firefox can be remote debugged by a second copy of Firefox. Here's how Simon explains it:
 
Unfortunately my attempts (via about:config) failed so far. But maybe someone else knows some more tricks? 
user_pref("devtools.chrome.enabled", true);
user_pref("devtools.debugger.remote-enabled", true);
user_pref("devtools.debugger.prompt-connection", false);
and then some sources claim you need to add --start-debugger-server and a port (this would obviously need to be implemented for sfos-browser...)

and finally you could connect using about:debugging from desktop to debug what’s happening on the phone.

This approach isn't something I've tried before and it's true that it may need some code changes, but they may not be significant changes. I've not had a chance to try this, but plan to give it a go some time over the next week if I have time. Thanks Simon for the really nice suggestion!

Going back to the curent situation, the local gdb debugger is still running so I'd better spend a little time today making use of it. Recall that yesterday we found that on ESR 91 a breakpoint on nsDisplayBackgroundImage::AppendBackgroundItemsToTop was triggered, whereas a breakpoint on nsDisplayBackgroundImage::GetInitData() wasn't.

These two are potentially significant because the latter can be found one above the former in the stack when we visit the HTML version of DuckDuckGo. Consequently, it could be that something is happening in this method to prevent the site from rendering.

This feels like a long shot, especially because the stack for the HTML version of the page is hardly likely to be equivalent to the stack for the JavaScript version of the page. Nevetheless, we can pin this down a bit more using an ESR 78 version of the browser. If we run this on the JavaScript version of the page, we might find there's some useful comparison to be made.

I've placed breakpoints on nsDisplayBackgroundImage::GetInitData() for both versions of the browser and have run them simultaneously. On ESR 91 I get a blank page, nothing rendered, and the breackpoint remains untriggered. In contrast to this on ESR 78 the breakpoint hits with the following backtrace: 
Thread 8 "GeckoWorkerThre" hit Breakpoint 1,
    nsDisplayBackgroundImage::GetInitData (aBuilder=aBuilder@entry=0x7fa6e2e630,
    aFrame=aFrame@entry=0x7f88744060, aLayer=aLayer@entry=0,
    aBackgroundRect=..., aBackgroundStyle=aBackgroundStyle@entry=0x7f89d21578)
    at layout/painting/nsDisplayList.cpp:3233
3233                                          ComputedStyle* aBackgroundStyle) {
(gdb) bt
#0  nsDisplayBackgroundImage::GetInitData (aBuilder=aBuilder@entry=0x7fa6e2e630,
    aFrame=aFrame@entry=0x7f88744060, aLayer=aLayer@entry=0, 
    aBackgroundRect=..., aBackgroundStyle=aBackgroundStyle@entry=0x7f89d21578)
    at layout/painting/nsDisplayList.cpp:3233
#1  0x0000007fbc1fab50 in nsDisplayBackgroundImage::AppendBackgroundItemsToTop
    (aBuilder=aBuilder@entry=0x7fa6e2e630, aFrame=aFrame@entry=0x7f88744060, 
    aBackgroundRect=..., aList=<optimized out>, 
    aAllowWillPaintBorderOptimization=aAllowWillPaintBorderOptimization@entry=true, 
    aComputedStyle=aComputedStyle@entry=0x0, aBackgroundOriginRect=..., 
    aSecondaryReferenceFrame=aSecondaryReferenceFrame@entry=0x0, 
    aAutoBuildingDisplayList=<optimized out>, aAutoBuildingDisplayList@entry=0x0)
    at layout/painting/nsDisplayList.cpp:3605
#2  0x0000007fbbffea94 in nsFrame::DisplayBackgroundUnconditional
    (this=this@entry=0x7f88744060, aBuilder=aBuilder@entry=0x7fa6e2e630, aLists=..., 
    aForceBackground=aForceBackground@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsRect.h:42
[...]
#74 0x0000007fb735e89c in ?? () from /lib64/libc.so.6
(gdb)
This is interesting because the second frame in the stack is for nsDisplayBackgroundImage::AppendBackgroundItemsToTop(). That matches what we were expecting before, and there's a good chance that DuckDuckGo will be serving the same data to ESR 78 as to the ESR 91 version of the renderer.

This strengthens my suspicion that there's something to be found inside AppendBackgroundItemsToTop(). Let's take a look at it.

So now I'm stepping through the code side-by-side using two phones; ESR 78 on the phone on the left and ESR 91 on the phone on the right. On my laptop display I have Qt Creator open with the ESR 78 copy of nsDisplayList.cpp on the left and the ESR 91 copy of the file on the right.

As I step through on both phones I'm comparing the source code line-by-line. They're not identical but close enough that I can keep track of them line-by-line and keep them broadly synchronised.

Eventually we hit this bit of code that's part of ESR 78, which is where the method returns: 
  if (!bg) {
    aList->AppendToTop(&bgItemList);
    return false;
  }
The ESR 91 version of this code looks like this: 
  if (!bg || !drawBackgroundImage) {
    if (!bgItemList.IsEmpty()) {
      aList->AppendToTop(&bgItemList);
      return AppendedBackgroundType::Background;
    }

    return AppendedBackgroundType::None;
  }
That's similar but not quite the same and in particular, bgItemList.mLength is zero meaning that !bgItemList.IsEmpty() is false. 
(gdb) p bgItemList.mLength
$6 = 0
Both methods return at this point, but on ESR 78 aList->AppendToTop() has been called whereas on ESR 91 it's been skipped. There are multiple reasons why this might be and it's hard to imagine this is the reason rendering is failing on ESR 91, but it's a possibility that I need to investigate further.

But also in both cases the method is returning before the call to GetInitData() and what I'm really after is a case where it's called on ESR 78 but not on ESR 91. To examine that I'm going to have to step through the method a few more times, maybe place a breakpoint closer to the call. And for that, unfortunately, I've run out of time for today; it'll have to be something I explore tomorrow.

So I guess the phones will have to be left stuck on a breakpoint overnight yet again.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
7 Jan 2024 : Day 131 #
I'm still stuck confused about why, specifically, the main DuckDuckGo search page doesn't render using ESR 91. As we saw yesterday, rotating the screen doesn't fix it. Copying the entire page and serving it from a different location works okay, which is the opposite of what I was hoping for and makes narrowing down the problem that much more difficult.

The only thing I can think to do now is crack open the debugger and see whether there's being any attempt to render anything of the site.

If the page is rendering, there's a good chance that some part of it will be an image. Consequently I've placed a breakpoint on nsImageRenderer::PrepareImage() and have set the browser running. Let's see if it hits. 
$ EMBED_CONSOLE=1 gdb sailfish-browser https://duckduckgo.com/
[...]
(gdb) b nsImageRenderer::PrepareImage
Function "nsImageRenderer::PrepareImage" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (nsImageRenderer::PrepareImage) pending.
(gdb) r
Starting program: /usr/bin/sailfish-browser 
[...]
An interesting result: there are no hits on the breakpoint at all when loading the default DuckDuckGo page. But when loading the HTML-only page, the breakpoint hits immedately: 
Thread 10 "GeckoWorkerThre" hit Breakpoint 1,
    mozilla::nsImageRenderer::PrepareImage (this=this@entry=0x7f9efb0848)
    at layout/painting/nsImageRenderer.cpp:66
66      bool nsImageRenderer::PrepareImage() {
In fact it hits, and continues hitting, while the page is displayed. Which is interesing: it suggests that for the JavaScript page no attempt is being made to render the page at all.

Here's the backtrace from the breakpoint hit with the HTML-only page. This could be useful, because it may allow us to place a breakpoint further up the stack to figure out at which point the render is getting blocked. That's assuming that the problem is in the render loop, which of course it may well not be.

The 69-frame backtrace is huge, so I'm just copying out the relevant parts of it below. Rendering involves recursive calls to BuildDisplayList() which seems to be one of the reasons these stacks get so large. 
Thread 10 "GeckoWorkerThre" hit Breakpoint 1, mozilla::nsImageRenderer::
    PrepareImage (this=this@entry=0x7f9efa0848)
    at layout/painting/nsImageRenderer.cpp:66
66      bool nsImageRenderer::PrepareImage() {
(gdb) bt
#0  mozilla::nsImageRenderer::PrepareImage (this=this@entry=0x7f9efa0848)
    at layout/painting/nsImageRenderer.cpp:66
#1  0x0000007fbc350c5c in nsCSSRendering::PrepareImageLayer
    (aPresContext=aPresContext@entry=0x7f8111dde0,
    aForFrame=aForFrame@entry=0x7f813062c0, 
    aFlags=<optimized out>, aBorderArea=..., aBGClipRect=..., aLayer=..., 
    aOutIsTransformedFixed=aOutIsTransformedFixed@entry=0x7f9efa0847)
    at layout/painting/nsCSSRendering.cpp:2976
#2  0x0000007fbc351254 in nsDisplayBackgroundImage::GetInitData
    (aBuilder=aBuilder@entry=0x7f9efa6268, aFrame=aFrame@entry=0x7f813062c0, 
    aLayer=aLayer@entry=1, aBackgroundRect=...,
    aBackgroundStyle=aBackgroundStyle@entry=0x7f8130f608)
    at layout/painting/nsDisplayList.cpp:3416
#3  0x0000007fbc3a97a4 in nsDisplayBackgroundImage::AppendBackgroundItemsToTop
    (aBuilder=0x7f9efa6268, aFrame=0x7f813062c0, aBackgroundRect=..., 
    aList=0x7f9efa11c8, aAllowWillPaintBorderOptimization=<optimized out>,
    aComputedStyle=<optimized out>, aBackgroundOriginRect=..., 
    aSecondaryReferenceFrame=0x0, aAutoBuildingDisplayList=0x0)
    at layout/painting/nsDisplayList.cpp:3794
#4  0x0000007fbc18c3d0 in nsIFrame::DisplayBackgroundUnconditional
    (this=this@entry=0x7f813062c0, aBuilder=aBuilder@entry=0x7f9efa6268,
    aLists=..., aForceBackground=aForceBackground@entry=false)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/nsRect.h:40
#5  0x0000007fbc191c90 in nsIFrame::DisplayBorderBackgroundOutline
    (this=this@entry=0x7f813062c0, aBuilder=aBuilder@entry=0x7f9efa6268,
    aLists=..., aForceBackground=aForceBackground@entry=false)
    at layout/generic/nsIFrame.cpp:2590
#6  0x0000007fbc16fd10 in nsBlockFrame::BuildDisplayList
    (this=0x7f813062c0, aBuilder=0x7f9efa6268, aLists=...)
    at layout/generic/nsBlockFrame.cpp:6963
#7  0x0000007fbc1c040c in nsIFrame::BuildDisplayListForChild
    (this=this@entry=0x7f81306200, aBuilder=aBuilder@entry=0x7f9efa6268, 
    aChild=aChild@entry=0x7f813062c0, aLists=..., aFlags=..., aFlags@entry=...)
    at layout/generic/nsIFrame.cpp:4278
#8  0x0000007fbc159ccc in DisplayLine (aBuilder=aBuilder@entry=0x7f9efa6268,
    aLine=..., aLineInLine=aLineInLine@entry=false, aLists=..., 
    aFrame=aFrame@entry=0x7f81306200, aTextOverflow=aTextOverflow@entry=0x0,
    aLineNumberForTextOverflow=aLineNumberForTextOverflow@entry=0, 
    aDepth=aDepth@entry=0, aDrawnLines=@0x7f9efa14fc: 127)
    at layout/generic/nsBlockFrame.cpp:6924
#9  0x0000007fbc170220 in nsBlockFrame::BuildDisplayList (this=0x7f81306200,
    aBuilder=0x7f9efa6268, aLists=...)
    at layout/generic/nsBlockFrame.cpp:7082
[...]
#33 0x0000007fbc1bce14 in nsIFrame::BuildDisplayListForStackingContext
    (this=this@entry=0x7f81304940, aBuilder=<optimized out>, 
    aBuilder@entry=0x7f9efa6268, aList=aList@entry=0x7f9efa8078,
    aCreatedContainerItem=aCreatedContainerItem@entry=0x0)
    at layout/generic/nsIFrame.cpp:3416
#34 0x0000007fbc12d7ac in nsLayoutUtils::PaintFrame
    (aRenderingContext=aRenderingContext@entry=0x0,
    aFrame=aFrame@entry=0x7f81304940, aDirtyRegion=...,
    aBackstop=aBackstop@entry=4294967295,
    aBuilderMode=aBuilderMode@entry=nsDisplayListBuilderMode::Painting,
    aFlags=aFlags@entry=(nsLayoutUtils::PaintFrameFlags::WidgetLayers |
    nsLayoutUtils::PaintFrameFlags::ExistingTransaction |
    nsLayoutUtils::PaintFrameFlags::NoComposite))
    at layout/base/nsLayoutUtils.cpp:3445
#35 0x0000007fbc0b9008 in mozilla::PresShell::Paint
    (this=this@entry=0x7f811424a0, aViewToPaint=aViewToPaint@entry=0x7e780077f0,
    aDirtyRegion=..., aFlags=aFlags@entry=mozilla::PaintFlags::PaintLayers)
    at layout/base/PresShell.cpp:6400
[...]
#69 0x0000007fb78b289c in ?? () from /lib64/libc.so.6
(gdb)
This breakpoint was set on nsImageRenderer::PrepareImage(). That hit on the HTML page but not on the JavaScript page. Let's try something further down the stack. At stack frame 34 we're inside nsLayoutUtils::PaintFrame(), so let's put a breakpoint on that and display the JavaScript version of the page to see what happens. 
(gdb) delete break 1
(gdb) b nsLayoutUtils::PaintFrame
Breakpoint 2 at 0x7fbc12ce1c: file layout/base/nsLayoutUtils.cpp, line 3144.
(gdb) c
Continuing.

Thread 10 "GeckoWorkerThre" hit Breakpoint 2, nsLayoutUtils::PaintFrame
    (aRenderingContext=aRenderingContext@entry=0x0,
    aFrame=aFrame@entry=0x7f809f3c00, 
    aDirtyRegion=..., aBackstop=aBackstop@entry=4294967295,
    aBuilderMode=aBuilderMode@entry=nsDisplayListBuilderMode::Painting, 
    aFlags=aFlags@entry=(nsLayoutUtils::PaintFrameFlags::WidgetLayers |
    nsLayoutUtils::PaintFrameFlags::ExistingTransaction |
    nsLayoutUtils::PaintFrameFlags::NoComposite))
    at layout/base/nsLayoutUtils.cpp:3144
3144                                       PaintFrameFlags aFlags) {
(gdb) bt
#0  nsLayoutUtils::PaintFrame (aRenderingContext=aRenderingContext@entry=0x0,
    aFrame=aFrame@entry=0x7f809f3c00, aDirtyRegion=..., 
    aBackstop=aBackstop@entry=4294967295,
    aBuilderMode=aBuilderMode@entry=nsDisplayListBuilderMode::Painting, 
    aFlags=aFlags@entry=(nsLayoutUtils::PaintFrameFlags::WidgetLayers |
    nsLayoutUtils::PaintFrameFlags::ExistingTransaction |
    nsLayoutUtils::PaintFrameFlags::NoComposite))
    at layout/base/nsLayoutUtils.cpp:3144
#1  0x0000007fbc0b9008 in mozilla::PresShell::Paint
    (this=this@entry=0x7f80456d90, aViewToPaint=aViewToPaint@entry=0x7f80b73520,
    aDirtyRegion=..., aFlags=aFlags@entry=mozilla::PaintFlags::PaintLayers)
    at layout/base/PresShell.cpp:6400
#2  0x0000007fbbef0ec8 in nsViewManager::ProcessPendingUpdatesPaint
    (this=this@entry=0x7f80b79520, aWidget=aWidget@entry=0x7f806e6f60)
    at ${PROJECT}/obj-build-mer-qt-xr/dist/include/mozilla/gfx/RectAbsolute.h:43
[...]
#30 0x0000007fb78b289c in ?? () from /lib64/libc.so.6
(gdb)
And we get a hit. That means that somewhere between the call to nsLayoutUtils::PaintFrame() and nsImageRenderer::PrepareImage something is changing.

I won't bore you with all of the steps, but I've followed the path up through the stack, setting breakpoints on each method call, and have found that the following breakpoint does hit for the standard rendering of the DuckDuckGo site: 
Thread 10 "GeckoWorkerThre" hit Breakpoint 6,
    nsDisplayBackgroundImage::AppendBackgroundItemsToTop (aBuilder=0x7f9efa6378,
    aFrame=0x7f81599428, aBackgroundRect=..., aList=0x7f9efa3f08,
    aAllowWillPaintBorderOptimization=true, aComputedStyle=0x0,
    aBackgroundOriginRect=..., aSecondaryReferenceFrame=0x0,
    aAutoBuildingDisplayList=0x0)
    at layout/painting/nsDisplayList.cpp:3632
3632            aAutoBuildingDisplayList) {
(gdb)
However, moving one up from the stack we find nsDisplayBackgroundImage::GetInitData() and this method doesn't trigger when we placed a breakpoint on it: 
(gdb) delete break 6
(gdb) break nsDisplayBackgroundImage::GetInitData
Breakpoint 7 at 0x7fbc3511d4: file layout/painting/nsDisplayList.cpp, line 3409.
(gdb) c
Continuing.
So that's between stack frame two and three of our original backtrace. This might suggest — although I'm not on firm ground here by any means — that the problem is potentially happening inside the call to AppendBackgroundItemsToTop(). On the other hand, that might just be a consequence of the two versions of the site having different structures. I'm going to step through the method to try to find out.

But unfortunately not today: it's late and this needs a bit more time, so I'll pick it up in the morning. This is actually quite a good place to pause, having a very clear direction to explore and pick up on tomorrow.

Let's see what tomorrow brings.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
6 Jan 2024 : Day 130 #
We didn't make much progress yesterday despite capturing all of the data flowing through the browser while visiting DuckDuckGo. While using ESR 91 the page just comes up completely blank, with no obvious error messages in the console output to indicate why.

I left things overnight to settle in my mind and by the morning I'd come up with a plan. So I suppose letting it settle was the right move.

But before getting in to that let me first thank PeperJohnny for sympathising with my testing plight. While of course I'm not happy to PeperJohnny suffers these frustrations as well, I am reassured by the knowledge that I'm not the only one! I share — and am encouraged by — PeperJohnny's belief that the reason can't hide itself forever!

Thanks also to hschmitt and lkraav for the useful comments about similar experiences rendering the Amazon Germany Web pages using the ESR 78 engine.

As it happens I also get this with Amazon UK using ESR 78. It's mighty unhelpful because pages appear briefly before blanking out, making the site next-to-useless. I wasn't aware of the portrait-to-landscape fix, which from the useful comments, I now find works very well. So thank you both for this nice input.

Unfortunately the problem with DuckDuckGo on ESR 91 seems to be different. Unlike the Amazon case the rendering doesn't even start, so the symptoms appear to be slightly different. Following the advice I did of course also try the portrait-to-landscape trick, but sadly this doesn't have any obvious effect.

One positive I can report is that Amazon no longer exhibits this annoying behaviour when using ESR 91. It does display other problematic behaviour, but fixing that will have to wait for a future investigation.

Thanks for all of the input, I always appreciate it, and am always pleased to follow up useful tips like these. When attempting to fix this kind of thing, ideas can be the scarcest of resources, so all suggestions are valuable.

So it looks like I still need a plan of attack. Continuing with the plan I cooked up overnight, unfortunately I don't have much time to execute it today, but maybe I can make a start on it. The approach I'm going to try is a "divide and conquer" approach. I've actually used it before when trying to fix other sites broken with earlier versions of the browser. Given this I'm not sure why it didn't occur to me last night; sometimes these things just take a little while to work their way out.

So my plan is to take a complete static copy of the DuckDuckGo HTML and store it on a server I have control over and can access from my development phones. I can then visit the site and, hopefully, the page will be similarly broken.

Having got to that point I can then start removing parts of the site to figure out which particular aspect of it is causing the rendering to fail. I call it "divide and conquer" because each time I'll take away a chunk of the code and see whether that solves the problem. If not, I'll remove a different chunk. Eventually something will have to render.

The tricky part here is getting an adequate copy of the site. If it's not close enough the issue won't manifest itself.

I've started off by exporting a full copy of the site using desktop Firefox. I started by taking a copy of the page while using the Responsive Design developer option to make the site believe it's running on a phone. But taking this site, posting it on a server and accessing it using ESR 91, I find this new copy of the site now loads perfectly.

So I tried a second time, this time capturing the full desktop site. Again, when I load this using ESR 91 on my phone it looks just fine, automatically displaying the mobile version.

This is all rather unhelpful.

Unfortunately it's now super-late here and I've still not got to the bottom of this. Maybe another night's sleep will help.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
5 Jan 2024 : Day 129 #
Yesterday we were looking at the User Agent strings, which I updated for ESR 91, but which we concluded wasn't the cause of the poorly rendering websites I've been experiencing.

One site that I'm particularly disappointed at being broken — and which I also mentioned yesterday — is DuckDuckGo. As this is currently my preferred search engine (although I admit I'm on the look out for a new one) it would be rather convenient for me personally if it was working well with the Sailfish Browser.

There is a workaround, in that DuckDuckGo already provides a non-JavaScript version of their site at https://duckduckgo.com/html. This site already works just fine with ESR 91 and I admit it's these kinds of considerate touches that keep me going back to use the site. But that doesn't detract from the fact that the standard JavaScript enabled site really ought to be working just fine with ESR 91.

So today I want to investigate further what the problem is. I'm a bit short on time today, so not expecting to get to the bottom of it, but it would be good to make a start.

It's also worth recalling that there were some problems with getting DuckDuckGo to work on previous versions of the browser. Digging through the git logs I eventually hit this, in the embedlite-components repository, which is likely what I'm thinking of: 
$ git log --grep "DuckDuckGo"
commit 04bf236f9a57bdd01d02d83f02e55b848a8ed1af (upstream/jb45659, origin/jb45659)
Author: David Llewellyn-Jones <david@flypig.co.uk>
Date:   Fri Aug 14 16:24:22 2020 +0000

    [embedlite-components] Ensure touch events also fire mouse clicks.
    Fixes JB#45659
    
    Gesture single taps were previously configured to send out mouse events
    only if they were preceded by a touchstart event, and only if
    preventDefault() wasn't applied to the event.
    
    Other browsers send out the event independent of this. The difference
    manifests itself if stopPropagation() is applied to the touch event,
    which supresses the mouse events on our browser, but not on others.
    
    For example, this meant that the input field of DuckDuckGo couldn't be
    focussed, and also prevented the Google Maps touch controls from
    working.
    
    This change alters things so that the mouse event is sent out even if
    single tap isn't preceeded by a touch event.
Please try to ignore the fact that back in 2020 I apparently didn't know how to spell either preceded or suppresses. I want to focus on the fact that today I'm experiencing something different on ESR 91: now the site just doesn't render at all.

When rendering the site, the only output in the console log is the following: 
[JavaScript Warning: "Cookie “_ga_NL3SRVXF92” will be soon rejected because it
    has the “SameSite” attribute set to “None” or an invalid value, without the
    “secure” attribute. To know more about the “SameSite“ attribute, read
    https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite"
    {file: "https://www.googletagmanager.com/gtag/js?id=G-NL3SRVXF92&l=dataLayer&cx=c"
    line: 307}]
That's a warning not an error and even if it was an error it doesn't look like it would be too serious. Not serious enough to prevent the site from rendering at any rate. So this doesn't help us much.

Given the complete lack of rendering it's hard to even know whether the correct site is being transferred or not: it may be that the browser is getting stuck on a redirect or something mundane like that.

To check this we can set the EMBED_CONSOLE environment variable to the value "network". This is a special value that, as well as printing out full debug information, also outputs everything textual that's sent between the network and the browser. This can generate crazy quantities of debug output, so it's almost never a good idea to use it. But in this case where we're just checking the contents of a single page being downloaded it can be really helpful.

Let's give it a go.

In all of the output below I've made quite significant abridgements to prevent the output from filling up multiple browser pages. I've tried to remove only uninteresting parts (mostly, the returned HTML code). 
$ EMBED_CONSOLE="network" MOZ_LOG="EmbedLite:5" sailfish-browser \
    https://duckduckgo.com/
[...]
[JavaScript Error: "Unexpected event profile-after-change" {file:
    "resource://gre/modules/URLQueryStrippingListService.jsm" line: 228}]
observe@resource://gre/modules/URLQueryStrippingListService.jsm:228:12
[...]
CONSOLE message:
OpenGL compositor Initialized Succesfully.
Version: OpenGL ES 3.2 V@415.0 (GIT@248cd04, I42b5383e2c, 1569430435)
    (Date:09/25/19)
Vendor: Qualcomm
Renderer: Adreno (TM) 610
FBO Texture Target: TEXTURE_2D
[Parent 11746: Unnamed thread 7bfc002670]: I/EmbedLite WARN: EmbedLite::virtual
    void* mozilla::embedlite::EmbedLitePuppetWidget::GetNativeData(uint32_t):127
    EmbedLitePuppetWidget::GetNativeData not implemented for this type
JSScript: ContextMenuHandler.js loaded
JSScript: SelectionPrototype.js loaded
JSScript: SelectionHandler.js loaded
JSScript: SelectAsyncHelper.js loaded
JSScript: FormAssistant.js loaded
JSScript: InputMethodHandler.js loaded
EmbedHelper init called
Available locales: en-US, fi, ru
Frame script: embedhelper.js loaded
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://duckduckgo.com/
    [ Request headers --------------------------------------- ]
        Host : duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101
            Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
        If-None-Match : "65959a92-471e"
        TE : trailers
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Wed, 03 Jan 2024 21:07:55 GMT
        content-type : text/html; charset=UTF-8
        content-length : 18206
        vary : Accept-Encoding
        etag : "65959a94-471e"
        content-encoding : br
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : default-src 'none' ; connect-src
            https://duckduckgo.com https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; manifest-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; media-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; script-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com 'unsafe-inline' 'unsafe-eval' ;
            font-src data:  https://duckduckgo.com https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; img-src data:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; style-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com 'unsafe-inline' ; object-src 'none' ;
            worker-src blob: ; child-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; frame-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; form-action  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; frame-ancestors 'self' ;
            base-uri 'self' ; block-all-mixed-content ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Wed, 03 Jan 2024 21:07:54 GMT
        cache-control : no-cache
        X-Firefox-Spdy : h2
    [ Document content -------------------------------------- ]
<!DOCTYPE html><html lang="en-US"><head><meta charSet="utf-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1,
    user-scalable=1 , viewport-fit=auto"/><link rel="preload"
    href="/static-assets/font/ProximaNova-RegIt-webfont.woff2" as="font"
    type="font/woff2" crossorigin="anonymous"/>
    [...]
    <script>window.onerror = function _onerror(msg, source, lineno, colno) {
  var url = "https://improving.duckduckgo.com/t/" + "static_err_global?" +
      Math.ceil(Math.random() * 1e7) + "&msg=" + encodeURIComponent(msg) +
      "&url=" + encodeURIComponent(source) + "&pathname=" +
      encodeURIComponent(window.location && window.location.pathname || "") +
      "&line=" + lineno + "&col=" + colno;

  try {   
    if (window.navigator.sendBeacon) {
      window.navigator.sendBeacon(url);
    } else {
      var img = document.createElement("img");
      img.src = url;
    }
  } catch (e) {// noop
  }
};</script>
[...]
        Document output truncated by 73144 bytes
    [ Document content ends --------------------------------- ]
JavaScript error: chrome://embedlite/content/embedhelper.js, line 259:
    TypeError: sessionHistory is null
CONSOLE message:
[JavaScript Error: "TypeError: sessionHistory is null"
    {file: "chrome://embedlite/content/embedhelper.js" line: 259}]
receiveMessage@chrome://embedlite/content/embedhelper.js:259:29
There's clearly plenty of document data being downloaded. I'm not sure what to make of this. It doesn't look like the site content is being held back; there's no obvious redirect. Frankly it all looks in order.

As a separate test I also tried turning off JavaScript to see how this might affect the rendering, but in that case the site was clever enough to notice and redirected the browser to https://html.duckduckgo.com/, which of course then rendered just fine.

We can compare the output with JavaScript disabled to that generated from the plain HTML version of the page. 
$ EMBED_CONSOLE="network" MOZ_LOG="EmbedLite:5" sailfish-browser \
    https://html.duckduckgo.com/
[...]
[W] unknown:0 - Unable to open bookmarks  "/home/defaultuser/.local/share/
    org.sailfishos/browser/bookmarks.json"
[...]
[Parent 14102: Unnamed thread 776c002670]: E/EmbedLite FUNC::static bool
    GeckoLoader::InitEmbedding(const char*):230 InitEmbedding successfully
CONSOLE message:
[JavaScript Error: "Unexpected event profile-after-change" {file: "resource://gre/modules/URLQueryStrippingListService.jsm" line: 228}]
observe@resource://gre/modules/URLQueryStrippingListService.jsm:228:12
[...]
JSScript: ContextMenuHandler.js loaded
JSScript: SelectionPrototype.js loaded
JSScript: SelectionHandler.js loaded
JSScript: SelectAsyncHelper.js loaded
JSScript: FormAssistant.js loaded
JSScript: InputMethodHandler.js loaded
EmbedHelper init called
Available locales: en-US, fi, ru
Frame script: embedhelper.js loaded
[ Request details ------------------------------------------- ]
    Request: GET status: 302 Found
    URL: https://html.duckduckgo.com/
[ Request details ------------------------------------------- ]
    Request: GET status: 200 OK
    URL: https://html.duckduckgo.com/html/
    [ Request headers --------------------------------------- ]
        Host : html.duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101
            Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Wed, 03 Jan 2024 21:31:43 GMT
        content-type : text/html
        content-length : 138
        location : https://html.duckduckgo.com/html/
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : default-src 'none' ; connect-src
            https://duckduckgo.com https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; manifest-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; media-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; script-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com 'unsafe-inline' 'unsafe-eval' ;
            font-src data:  https://duckduckgo.com https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; img-src data:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; style-src  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com 'unsafe-inline' ; object-src 'none' ;
            worker-src blob: ; child-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; frame-src blob:  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; form-action  https://duckduckgo.com
            https://*.duckduckgo.com
            https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/
            https://spreadprivacy.com ; frame-ancestors 'self' ;
            base-uri 'self' ; block-all-mixed-content ;
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Thu, 02 Jan 2025 21:31:43 GMT
        cache-control : max-age=31536000
        x-duckduckgo-locale : en_GB
        X-Firefox-Spdy : h2
    [ Request headers --------------------------------------- ]
        Host : html.duckduckgo.com
        User-Agent : Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101
            Firefox/91.0
        Accept : text/html,application/xhtml+xml,application/xml;q=0.9,
            image/webp,*/*;q=0.8
        Accept-Language : en-GB,en;q=0.5
        Accept-Encoding : gzip, deflate, br
        Connection : keep-alive
        Upgrade-Insecure-Requests : 1
        Sec-Fetch-Dest : document
        Sec-Fetch-Mode : navigate
        Sec-Fetch-Site : cross-site
        TE : trailers
    [ Response headers -------------------------------------- ]
        server : nginx
        date : Wed, 03 Jan 2024 21:31:43 GMT
        content-type : text/html; charset=UTF-8
        vary : Accept-Encoding
        server-timing : total;dur=14;desc="Backend Total"
        strict-transport-security : max-age=31536000
        permissions-policy : interest-cohort=()
        content-security-policy : default-src 'none' ; connect-src
            https://duckduckgo.com https://*.duckduckgo.com
[...]
        x-frame-options : SAMEORIGIN
        x-xss-protection : 1;mode=block
        x-content-type-options : nosniff
        referrer-policy : origin
        expect-ct : max-age=0
        expires : Wed, 03 Jan 2024 21:31:44 GMT
        cache-control : max-age=1
        x-duckduckgo-locale : en_GB
        content-encoding : br
        X-Firefox-Spdy : h2
    [ Document content -------------------------------------- ]
        Document output skipped, content-type non-text or unknown
    [ Document content ends --------------------------------- ]
    [ Document content -------------------------------------- ]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<!--[if IE 6]><html class="ie6" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->
<!--[if IE 7]><html class="lt-ie8 lt-ie9" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->
<!--[if IE 8]><html class="lt-ie9" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->
<!--[if gt IE 8]><!--><html xmlns="http://www.w3.org/1999/xhtml"><!--<![endif]-->
<head>
  <link rel="canonical" href="https://duckduckgo.com/" />
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0,
      maximum-scale=3.0, user-scalable=1" />
  <meta name="referrer" content="origin" />
  <title>DuckDuckGo</title>
  <link title="DuckDuckGo (HTML)" type="application/opensearchdescription+xml"
      rel="search" href="//duckduckgo.com/opensearch_html_v2.xml" />
  <link rel="icon" href="//duckduckgo.com/favicon.ico" type="image/x-icon" />
  <link rel="apple-touch-icon" href="//duckduckgo.com/assets/logo_icon128.v101.png" />
  <link rel="image_src" href="//duckduckgo.com/assets/logo_homepage.normal.v101.png" />
  <link rel="stylesheet" media="handheld, all" href="//duckduckgo.com/dist/h.aeda52882d97098ab9ec.css" type="text/css"/>
</head>

<body class="body--home body--html">
[...]
</html>

    [ Document content ends --------------------------------- ]
JavaScript error: chrome://embedlite/content/embedhelper.js, line 259:
    TypeError: sessionHistory is null
CONSOLE message:
[JavaScript Error: "TypeError: sessionHistory is null"
    {file: "chrome://embedlite/content/embedhelper.js" line: 259}]
receiveMessage@chrome://embedlite/content/embedhelper.js:259:29

JavaScript error: resource://gre/modules/LoginManagerChild.jsm, line 541:
    NotFoundError: WindowGlobalChild.getActor: No such JSWindowActor 'LoginManager'
CONSOLE message:
[JavaScript Error: "NotFoundError: WindowGlobalChild.getActor: No such
    JSWindowActor 'LoginManager'"
    {file: "resource://gre/modules/LoginManagerChild.jsm" line: 541}]
forWindow@resource://gre/modules/LoginManagerChild.jsm:541:27
handleEvent@chrome://embedlite/content/embedhelper.js:433:29
The main difference in this second case is that now we have a 302 "Found" redirect returned from the site, which then takes us to the JavaScript-free HTML version. Other than this, comparing the two unfortunately I don't see any obvious signs of why one might be working but the other not.

So I'm a bit stumped. At this point my head is spinning a bit; it's late here. But I'm frustrated and don't want to stop just yet. Maybe there's another — simpler — site that exhibits the same problem? If I could find something like that, it might help.

So to try to take this further I tested all of the pages from the Sailfish Browser test suite. I thought maybe something might show up as broken there that, if fixed, might also fix the DuckDuckGo page.

I was surprised to discover that most of these pages already work as expecting, including screen taps, SVGs and (some) videos. Audio wasn't working, neither was the file picker and perhaps most problematic the back button is also broken (although, weirdly, the page history seems to be working just fine).

So several things to fix, but none of the pages failed to render at all. So this also brought me no closer to figuring out what the problem might be with DuckDuckGo.

It's been a day of testing today it seems, rather than fixing. If I'm honest I'm a bit stumped and very frustrated right now. Maybe I need to sleep on it to try to think of another angle to approach it from.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
4 Jan 2024 : Day 128 #
It's the big one today! Finally we reached 27 days of Gecko development work. Looking at how things are progressing, it's looking hopeful that the work may be completed before we hit the 28-th days of work. But I'm not willing to make any more of a prediction than that!

Before moving on to coding, let's dwell on this a little further. I've been spending, I think (I've not actually been measuring this) between two to three hours working on this a day. Probably half that time is spent coding and the other half writing these posts. So that means 90 minutes of coding time spent on gecko for each of the 27 days.

That makes 192 hours of work, or 24 work days (eight hours a day), or nearly five weeks of "Full Time Equivalent" work. During holidays I've probably spent a bit more than this, so let's call it seven weeks.

That might sound like a lot, but in practice it's not much time at all when it comes to software development. I can imagine progress probably looks very slow to anyone reading this, but unfortunately it's more a product of the fact I can only spend some limited amount of time on this per day. I do wish there were more hours in the day to work with!

Alright, let's waste no more time on this side discussion then and get straight back to coding.

It's finally time to move on from printing, as promised, to User Agent string testing. This falls under Issue #1052 which I've now assigned to myself as my next task.

Let's get some bearings. The magic for User Agent fixes happens inside a file called ua-update.json that can be found inside your browser profile folder. It's a JSON file but un-prettified into a single line which makes it a little tricky to read. We can make it a little clearer by re-prettifying it. I'm just putting the first few lines here to give an idea about the contents, but feel free to run this on your own phone if you'd like to see the full list of fixes. 
$ python3 -m json.tool ~/.local/share/org.sailfishos/browser/.mozilla/ua-update.json
{
    "msn.com": "Mozilla/5.0 (Android 8.1.0; Mobile; rv:78.0) Gecko/78.0 Firefox/78.0",
    "r7.com": "Mozilla/5.0 (Android; Mobile; rv:78.0) Gecko/78.0 Firefox/78.0",
    "baidu.com": "Mozilla/5.0 (Android 8.1.0; Mobile; rv:78.0) Gecko/78.0 Firefox/78.0",
    "bing.com": "Mozilla/5.0 (Android; Mobile; rv:78.0) Gecko/78.0 Firefox/78.0",
    "kuruc.info": "Mozilla/5.0 (Android; Mobile; rv:78.0) Gecko/78.0 Firefox/78.0",
[...]
}
This file is generated using a process, so represents the end result of that process. The processing step happens during the sailfish-browser build and the raw pieces can be found in the data folder: 
$ tree data
data
├── 70-browser.conf
├── prefs.js
├── preprocess-ua-update-json
├── README
├── ua
│   ├── 38.8.0
│   │   └── ua-update.json
│   ├── 45.9.1
│   │   └── ua-update.json
│   ├── 52.9.1
│   │   └── ua-update.json
│   ├── 60.0
│   │   └── ua-update.json
│   ├── 60.9.1
│   │   └── ua-update.json
│   └── 78.0
│       └── ua-update.json
├── ua-update.json
└── ua-update.json.in

7 directories, 12 files
There are also some helpful instructions to be found in this folder: 
$ cat data/README 
How to update ua-update.json
============================

0) git checkout next
1) Edit ua-update.json.in
2) Run preprocess-ua-update-json
3) Copy preprocessed ua-update.json to ua/<engine-version>
4) git commit ua-update.json.in ua/<engine-version>/ua-update.json
5) Create merge request
Before we get into these steps let's first find out what User Agent string is actually being used when we visit a Web site. The easiest way to do this is to actually go to a site that echoes the User Agent back to us.
 
DuckDuckGo showing the original and overridden User Agent strings; Bing and Google don't show the string

On DuckDuckGo the User Agent string is announced as "Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Firefox/91.0)". That doesn't look too bad. But it's also not one of the sites we currently have an override set for.

Neither Bing nor Google, both of which are in the list, do the courtesy of echoing back the user agent when you search on it and none of the other sites in the list look promising for doing this either. So I'm going to hack a user agent string in for DuckDuckGo to see whether or not we get any results.

I'll do this by just switching the msn.com string to duckduckgo.com instead. 
$ cd ~/.local/share/org.sailfishos/browser/.mozilla/
$ cp ua-update.json ua-update.json.bak
$ sed -i -e 's/msn\.com/duckduckgo\.com/g' ua-update.json
Now when we visit the site we get a newly updated User Agent string: "Mozilla/5.0 (Android 8.1.0; Mobile; rv:78.0) Gecko/78.0 Firefox/78.8".

We can conclude that the ua-update.json file is working as expected, but we will need to update it to reflect the new rendering engine version. Before proceeding I'm going to undo the change I just made. It's not like it really matters on my development phone, but it helps keep things neat and tidy. 
$ mv ua-update.json.bak ua-update.json
Back to the sailfish-browser repository and I'll do a search-and-replace on the references to "78.0" and replace them with "91.0". The file may need a little more refinement based on how the rendering engine works with different sites, but this simple change should give us a more solid base to build on. 
$ pushd data
$ sed -i -e 's/78\.0/91\.0/g' ua-update.json.in 
$ git diff | head -n 10
diff --git a/data/ua-update.json.in b/data/ua-update.json.in
index be4352e6..584720c0 100644
--- a/data/ua-update.json.in
+++ b/data/ua-update.json.in
@@ -1,116 +1,116 @@
 // Everything after the first // on a line will be removed by the preproccesor.
 // Send these sites a custom user-agent. Bugs should be included with an entry.
 {
-  // Ref: "Mozilla/5.0 (Mobile; rv:78.0) Gecko/78.0 Firefox/78.0"
+  // Ref: "Mozilla/5.0 (Mobile; rv:91.0) Gecko/91.0 Firefox/91.0"
That looks pretty healthy. Let's complete the rest of the steps as suggested in the README. 
$ ./preprocess-ua-update-json 
[sailfishos-esr91 e00b4335] [user-agent] Update preprocessed user agent overrides
 1 file changed, 89 insertions(+), 89 deletions(-)
 rewrite data/ua-update.json (98%)
$ gedit ua-update.json
$ ls ua
38.8.0  45.9.1  52.9.1  60.0  60.9.1  78.0
$ mkdir ua/91.0
$ cp ua-update.json ua/91.0/
$ git add ua-update.json.in ua/91.0/ua-update.json
$ git commit -m "[sailfish-browser] Update user agent overrides for ESR 91.0"
$ popd
So I've now built and installed the updated sailfish-browser package. Checking the ua-update.json file on my phone I can see that the file hasn't changed after the installation. However, after deleting the ua-agent-json file in the profile folder and re-running the browser, the changes have now taken root. Great.

Although that's updated the overrides, unfortunately it hasn't fixed some of the broken sites such as DuckDuckGo.com, which still just displays a blank page. My recollection is that this happened with ESR 78 too, but there must have been a different reason for it. There may even be an answer in amongst the patches. This will have to be my next task for tomorrow.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
3 Jan 2024 : Day 127 #
As we rounded things off yesterday I was heading into a new task. My plan is to look into the User Agent string functionality of ESR 91. My hunch is that the overrides aren't working correctly in the updated build, although I'm not yet sure whether this is the case or why it might be the case. But I'm hoping that this is the reason for some sites currently working poorly with the updated browser engine.

But before I do that I want to try to address an issue raised by rob_kouw on the Sailfish forum. Or, to be more accurate, raised by Rob's wife. Here's Rob's comment:
 
I just read this morning’s post where you solved the pdf printing (hurray!) I remarked to my wife about the digging into this long, long list of threads: where to start?

After some more explanation she yells: “Who wants hidden browser tabs?”

And in fact, couldn’t these be misused at some point?


Apart from the fact that I'm chuffed that Rob is taking the time to read these posts, it's also a very good point: are these hidden browser tabs I've spent the last couple of weeks implementing going to be safe? Wouldn't it be a bad thing if websites have the ability to spawn hidden tabs without the user realising it?

I also want to acknowledge the useful input I received privately from thigg on this topic as well. These comments combined have provided really useful input when considering the final architecture.

So when we consider the security of these hidden tabs my claim is that it's safe because although there is now functionality to hide tabs, there is in fact only one way that this can happen and that's as a direct consequence of the user selecting the "Save web page to PDF" option in the Sailfish Browser menu. There's no other way it can happen and especially not through some malicious Web site creating hidden tabs without the user's consent.

But right now this is just a hypothesis. It deserves exploring in more depth.

Let's start at the sharp end. The tabs are hidden or shown based on the value stored in the hidden role in the DeclarativeTabModel class: 
QVariant DeclarativeTabModel::data(const QModelIndex & index, int role) const {
    if (index.row() < 0 || index.row() >= m_tabs.count())
        return QVariant();

    const Tab &tab = m_tabs.at(index.row());
    if (role == ThumbPathRole) {
        return tab.thumbnailPath();
    } else if (role == TitleRole) {
        return tab.title();
    } else if (role == UrlRole) {
        return tab.url();
    } else if (role == ActiveRole) {
        return tab.tabId() == m_activeTabId;
    } else if (role == TabIdRole) {
        return tab.tabId();
    } else if (role == DesktopModeRole) {
        return tab.desktopMode();
    } else if (role == HiddenRole) {
        return tab.hidden();
    }
    return QVariant();
}
This hidden value is exposed in the QML, but it's not exposed to the JavaScript of the page in the tab and it's also not a value that can be set after the page has been created: it's essentially constant.

As we can see from the last few lines, the hidden state comes from the Tab class. The only way to set it is when the tab is instantiated: 
Tab::Tab(int tabId, const QString &url, const QString &title,
    const QString &thumbPath, const bool hidden)
    : m_tabId(tabId)
    , m_requestedUrl(url)
    , m_url(url)
    , m_title(title)
    , m_thumbPath(thumbPath)
    , m_desktopMode(false)
    , m_hidden(hidden)
    , m_browsingContext(0)
    , m_parentId(0)
{
}
As we've seen in previous posts, the route that the hidden flag takes to get to this point is circuitous at best. But it all goes through only C++ code in the EmbedLite portion of the Sailfish gecko-dev project. None of this is exposed, either for reading or writing, to JavaScript. And it all leads back to this one location: 
NS_IMETHODIMP
WindowCreator::CreateChromeWindow(nsIWebBrowserChrome *aParent,
                                  uint32_t aChromeFlags,
                                  nsIOpenWindowInfo *aOpenWindowInfo,
                                  bool *aCancel,
                                  nsIWebBrowserChrome **_retval)
{
[...]
  const bool isForPrinting = aOpenWindowInfo->GetIsForPrinting();

  mChild->CreateWindow(parentID, reinterpret_cast<uintptr_t>(parentBrowsingContext.get()), aChromeFlags, isForPrinting, &createdID, aCancel);
This requires just a little explanation: it's the isForPrinting flag which becomes the hidden flag. This point here is the only route to the flag being set. But that's not enough to give us the confidence that there's only one way to control this flag. We also need to confirm how the isForPrinting flag gets set.

There are two places this gets called from, both of them in WindowWatcher.cpp. The first: 
NS_IMETHODIMP
nsWindowWatcher::OpenWindowWithRemoteTab(nsIRemoteTab* aRemoteTab,
                                         const nsACString& aFeatures,
                                         bool aCalledFromJS,
                                         float aOpenerFullZoom,
                                         nsIOpenWindowInfo* aOpenWindowInfo,
                                         nsIRemoteTab** aResult) {
And the second: 
nsresult nsWindowWatcher::OpenWindowInternal(
    mozIDOMWindowProxy* aParent, const nsACString& aUrl,
    const nsACString& aName, const nsACString& aFeatures, bool aCalledFromJS,
    bool aDialog, bool aNavigate, nsIArray* aArgv, bool aIsPopupSpam,
    bool aForceNoOpener, bool aForceNoReferrer, PrintKind aPrintKind,
    nsDocShellLoadState* aLoadState, BrowsingContext** aResult) {
[...]
For the first, the crucial data going in to this method is aOpenWindowInfo. For the second the flag gets set like this: 
    openWindowInfo->mIsForPrinting = aPrintKind != PRINT_NONE;
Consequently the crucial value going in is aPrintKind.

Let's now follow back the first of these. It comes via ContentParent::CommonCreateWindow() where it's passed in as the aForPrinting parameter like this: 
mozilla::ipc::IPCResult ContentParent::CommonCreateWindow(
    PBrowserParent* aThisTab, BrowsingContext* aParent, bool aSetOpener,
    const uint32_t& aChromeFlags, const bool& aCalledFromJS,
    const bool& aWidthSpecified, const bool& aForPrinting,
    const bool& aForWindowDotPrint, nsIURI* aURIToLoad,
    const nsCString& aFeatures, const float& aFullZoom,
    BrowserParent* aNextRemoteBrowser, const nsString& aName, nsresult& aResult,
    nsCOMPtr<nsIRemoteTab>& aNewRemoteTab, bool* aWindowIsNew,
    int32_t& aOpenLocation, nsIPrincipal* aTriggeringPrincipal,
    nsIReferrerInfo* aReferrerInfo, bool aLoadURI,
    nsIContentSecurityPolicy* aCsp, const OriginAttributes& aOriginAttributes) {
[...]
  openInfo->mIsForPrinting = aForPrinting;
[...]
  aResult = pwwatch->OpenWindowWithRemoteTab(thisBrowserHost, aFeatures,
                                             aCalledFromJS, aFullZoom, openInfo,
                                             getter_AddRefs(aNewRemoteTab));
[...]
Where this is called, in one case we have a hard coded false value passed in for isForPrinting: 
  mozilla::ipc::IPCResult ipcResult = CommonCreateWindow(
      aThisTab, parent, /* aSetOpener = */ false, aChromeFlags, aCalledFromJS,
      aWidthSpecified, /* aForPrinting = */ false,
      /* aForPrintPreview = */ false, aURIToLoad, aFeatures, aFullZoom,
      /* aNextRemoteBrowser = */ nullptr, aName, rv, newRemoteTab, &windowIsNew,
      openLocation, aTriggeringPrincipal, aReferrerInfo,
      /* aLoadUri = */ true, aCsp, aOriginAttributes);
In the other case the value is triggered via a call to ContentParent::RecvCreateWindow(). Following this back via SendCreateWindow() we find it eventually gets back to a call to nsWindowWatcher::OpenWindowInternal(). In other words, we find ourselves in the same place as the other path.

So let's move over to this path. We want to follow back to see what calls nsWindowWatcher::OpenWindowInternal() and in particular the value passed in for aPrintKind.

There are several ways this method can be called, all from inside nsWindowWatcher.cpp. One of them straightforwardly sets aPrintKind to PRINT_NONE: 
  MOZ_TRY(OpenWindowInternal(aParent, aUrl, aName, aFeatures,
                             /* calledFromJS = */ false, dialog,
                             /* navigate = */ true, argv,
                             /* aIsPopupSpam = */ false,
                             /* aForceNoOpener = */ false,
                             /* aForceNoReferrer = */ false, PRINT_NONE,
                             /* aLoadState = */ nullptr, getter_AddRefs(bc)));
The others both end up getting called in from two separate places in nsGlobalWindowOuter.cpp: 
$ grep -rIn "OpenWindow2" * --include="*.cpp"
gecko-dev/dom/base/nsGlobalWindowOuter.cpp:7134:
    rv = pwwatch->OpenWindow2(this, url, name, options,
gecko-dev/dom/base/nsGlobalWindowOuter.cpp:7154:
    rv = pwwatch->OpenWindow2(this, url, name, options,
gecko-dev/toolkit/components/windowwatcher/nsWindowWatcher.cpp:353:
    nsWindowWatcher::OpenWindow2(mozIDOMWindowProxy* aParent,
Both of these have the aPrintKind parameter set like this: 
  const auto wwPrintKind = [&] {
    switch (aPrintKind) {
      case PrintKind::None:
        return nsPIWindowWatcher::PRINT_NONE;
      case PrintKind::InternalPrint:
        return nsPIWindowWatcher::PRINT_INTERNAL;
      case PrintKind::WindowDotPrint:
        return nsPIWindowWatcher::PRINT_WINDOW_DOT_PRINT;
    }
    MOZ_ASSERT_UNREACHABLE("Wat");
    return nsPIWindowWatcher::PRINT_NONE;
  }();
In all cases the crucial aPrintKind value seen here is passed in as a parameter to nsGlobalWindowOuter::OpenInternal() which, in four out of five cases look like this: 
  return OpenInternal(aUrl, aName, aOptions,
                      true,                     // aDialog
                      false,                    // aContentModal
                      true,                     // aCalledNoScript
                      false,                    // aDoJSFixups
                      true,                     // aNavigate
                      nullptr, aExtraArgument,  // Arguments
                      nullptr,                  // aLoadState
                      false,                    // aForceNoOpener
                      PrintKind::None, _retval);
In these cases we can see it passed in explicitly as PrintKind::None. The exception is this call here: 
      auto printKind = aForWindowDotPrint == IsForWindowDotPrint::Yes
                           ? PrintKind::WindowDotPrint
                           : PrintKind::InternalPrint;
      aError = OpenInternal(u""_ns, u""_ns, u""_ns,
                            false,             // aDialog
                            false,             // aContentModal
                            true,              // aCalledNoScript
                            false,             // aDoJSFixups
                            true,              // aNavigate
                            nullptr, nullptr,  // No args
                            nullptr,           // aLoadState
                            false,             // aForceNoOpener
                            printKind, getter_AddRefs(bc));
Crucially, this route is found inside the following method: 
Nullable<WindowProxyHolder> nsGlobalWindowOuter::Print(
    nsIPrintSettings* aPrintSettings, nsIWebProgressListener* aListener,
    nsIDocShell* aDocShellToCloneInto, IsPreview aIsPreview,
    IsForWindowDotPrint aForWindowDotPrint,
    PrintPreviewResolver&& aPrintPreviewCallback, ErrorResult& aError) {
[...]
In other words, only a call to Print() will result in a value other than PRINT_NONE being passed in for aPrintKind and consequently the only way isForPrinting can be set to true.

We've followed all of the paths from beginning to end and hopefully it's clear that only this printing route will end up generating any hidden tabs. That's what we need. But in the future we'll also have to stay vigilant to the fact that this is possible; it's important we don't open any paths that allow it to be controlled by JavaScript coming from external sites.

My plan was to continue on today to looking into the User Agent strings, but this is already rather a long post so I'll leave it there for today. It remains to thank rob_kouw once again for the excellent question, as well as the useful and always-appreciated input from thigg.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
2 Jan 2024 : Day 126 #
We're finally reaching the end of the "Print web page to PDF" saga. Yesterday I re-implemented the JavaScript changes that were in gecko-dev as part of the embedlite-components package. Today I'm going to commit all of the changes so that they're available in the remote repositories as well as my local ones.

Going over the changes I previously made to the gecko-dev code to support the functionality I'm happy to see that all of them are now unnecessary. That means I can completely remove the last two commits. 
$ git checkout -b temp-FIREFOX_ESR_91_9_X_RELBRANCH_patches
$ git checkout FIREFOX_ESR_91_9_X_RELBRANCH_patches
$ git log -5 --oneline

2fb912372c04 (HEAD -> FIREFOX_ESR_91_9_X_RELBRANCH_patches,
    temp-FIREFOX_ESR_91_9_X_RELBRANCH_patches)
    Call Print from the CanonicalBrowingContext
26259399358f Reintroduce PDF printing code
e035d6ff3a78 (origin/FIREFOX_ESR_91_9_X_RELBRANCH_patches)
    Add embedlite static prefs
5ae7644719f8 Allow file scheme when loading OpenSearch providers
1fe7c56c7363 Supress URLQueryStrippingListService.jsm error

$ git reset --hard HEAD~~
HEAD is now at e035d6ff3a78 Add embedlite static prefs
$ git log -5 --oneline

e035d6ff3a78 (HEAD -> FIREFOX_ESR_91_9_X_RELBRANCH_patches,
    origin/FIREFOX_ESR_91_9_X_RELBRANCH_patches)
    Add embedlite static prefs
5ae7644719f8 Allow file scheme when loading OpenSearch providers
1fe7c56c7363 Supress URLQueryStrippingListService.jsm error
80fa227aee92 [sailfishos][gecko] Fix gfxPlatform::AsyncPanZoomEnabled
    for embedlite. JB#50863
b58093d23fb2 Add patch to fix 32-bit builds
These commits would have had to be turned into one or maybe two patches to the upstream code, so the fact I can remove them completely leaves me with a warm feeling inside.

Almost all of these changes are to the JavaScript code, so wouldn't require a rebuild of gecko-dev to apply them on my device in themselves. However there is one change to the nsIWebBrowserPrint interface which will have a knock on effect to the compiled (C++ and Rust) code as well, which is this one: 
@@ -98,7 +98,7 @@ interface nsIWebBrowserPrint : nsISupports
    * @param aWPListener - is updated during the print
    * @return void
    */
-  [noscript] void print(in nsIPrintSettings aThePrintSettings,
                         in nsIWebProgressListener aWPListener);
+  void print(in nsIPrintSettings aThePrintSettings,
 
   /**
Since we've shifted from calling print() on the window to calling print() on the canonical browsing context, this change shouldn't be needed any more. But to check that this really is the case I'm going to have to do a full rebuild. Having reverted the changes above I can therefore kick things off: 
$ sfdk -c no-fix-version build -d -p --with git_workaround
While this is building I've also created the issue that we considered yesterday as well. It's now Issue #1051: "Fix hang when calling window.setBrowserCover()".

I also created another issue that I've been noticing during testing of the ESR 91 code. There are various Web pages which don't work as well as expected, meaning that they don't work as well on ESR 91 as they do on ESR 78. That's not good for the end user. Previous versions of the browser have had similar problems and invariably the issue relates to what the site is serving to the browser. If the server doesn't recognise the User Agent string it will often send a different version of the page. Often these versions are broken or at least don't work well with the gecko rendering engine. It's pretty astonishing that this is still necessary in a world where Web standards are so much more clearly defined than they were even a decade ago, but there it is.

I've not yet looked into it but I have a suspicion that the reason some of these sites may not be working is that the Sailfish Browser's user agent string handling code is no longer sending the "adjusted" user agent strings to the sites that need them. So I've created Issue 1052 for looking in to this and trying to fix it if there's something to fix.

The gecko-dev build has completed (it took most of the day, but at least not all of the day). I'm now rebuilding the packages that depend on it so that I can then test them out.

It's almost the full suite: gecko-dev, qtmozembed-qt5, embedlite-components and sailfish-browser. The only package I've not rebuilt is sailfish-components-webview. I've not yet made any changes to that at all since starting this ESR 91 work.

All built. All uploaded. 
$ devel-su rpm -U --force xulrunner-qt5-91.*.rpm \
    xulrunner-qt5-debugsource-91.*.rpm xulrunner-qt5-debuginfo-91.*.rpm \
    xulrunner-qt5-misc-91.*.rpm qtmozembed-qt5-1.*.rpm \
    qtmozembed-qt5-debuginfo-1.*.rpm qtmozembed-qt5-debugsource-1.*.rpm \
    sailfish-browser-2.*.rpm sailfish-browser-debuginfo-2.*.rpm \
    sailfish-browser-debugsource-2.*.rpm sailfish-browser-settings-2.*.rpm \
    embedlite-components-qt5-1.*.rpm \
    embedlite-components-qt5-debuginfo-1.*.rpm \
    embedlite-components-qt5-debugsource-1.*.rpm
All installed.

Now to test the changes... and it doesn't work. That's no fun.

But as soon as I look at the code it's clear what the problem is. Here's the error: 
[JavaScript Error: "aSerializable.url is undefined"
    {file: "resource://gre/modules/DownloadCore.jsm" line: 1496}]
DownloadSource.fromSerializable@resource://gre/modules/DownloadCore.jsm:1496:5
Download.fromSerializable@resource://gre/modules/DownloadCore.jsm:1282:38
D_createDownload@resource://gre/modules/Downloads.jsm:108:39
DownloadPDFSaver2.fromSerializable@file:///usr/lib64/mozembedlite/components/
    EmbedliteDownloadManager.js:420:34
observe/<@file:///usr/lib64/mozembedlite/components/
    EmbedliteDownloadManager.js:272:56
Apart from the error in DownloadCore.jsm I'm also a little perturbed at the reference to DownloadPDFSaver2. While I had two versions of DownloadPDFSaver in the code at the same time I did call one of them DownloadPDFSaver2, but subsequently switched all references to just use DownloadPDFSaver (or so I thought) once I'd removed the unused version. So that really shouldn't be appearing at all.

Let's find out why.

First the reference to DownloadPDFSaver2. When I check the actual code on device it certainly does still reference this, but that's different to the code I have on my laptop. I must have made a mistake when building the package. I've rebuilt it, re-uploaded it and reinstalled it. Now when I run it I get this: 
[JavaScript Error: "aSerializable.url is undefined"
    {file: "resource://gre/modules/DownloadCore.jsm" line: 1496}]
DownloadSource.fromSerializable@resource://gre/modules/DownloadCore.jsm:1496:5
Download.fromSerializable@resource://gre/modules/DownloadCore.jsm:1282:38
D_createDownload@resource://gre/modules/Downloads.jsm:108:39
DownloadPDFSaver.createDownload@file:///usr/lib64/mozembedlite/components/
    EmbedliteDownloadManager.js:426:34
observe/<@file:///usr/lib64/mozembedlite/components/
    EmbedliteDownloadManager.js:272:55
That looks more healthy. Now for the main error. It looks to me like this is due to the removal of the following from DownloadCore.jsm: 
@@ -1499,12 +1491,6 @@ DownloadSource.fromSerializable = function(aSerializable) {
     source.url = aSerializable.toString();
   } else if (aSerializable instanceof Ci.nsIURI) {
     source.url = aSerializable.spec;
-  } else if (aSerializable instanceof Ci.nsIDOMWindow) {
-    source.url = aSerializable.location.href;
-    source.isPrivate = PrivateBrowsingUtils.isContentWindowPrivate(
-      aSerializable
-    );
-    source.windowRef = Cu.getWeakReference(aSerializable);
   } else {
     // Convert String objects to primitive strings at this point.
     source.url = aSerializable.url.toString();
In the new code in EmbedliteDownloadManager.js we send in the source like this: 
  let download = await DownloadPDFSaver.createDownload({
    source: Services.ww.activeWindow,
    target: data.to
  });
But now that I removed the code from DownloadCore.jsm it no longer knows how to handle this Services.ww.activeWindow value. I should be handling that in EmbedliteDownloadManager.js instead. At present the code that passes this value on looks like this: 
DownloadPDFSaver.createDownload = async function(aProperties) {
  let download = await Downloads.createDownload({
    source: aProperties.source,
    target: aProperties.target,
    contentType: "application/pdf"
  });
So I should be able to fix this by handling it more judiciously. Let's try this: 
DownloadPDFSaver.createDownload = async function(aProperties) {
  let download = await Downloads.createDownload({
    source: aProperties.source.location.href,
    target: aProperties.target,
    contentType: "application/pdf"
  });
  download.source.isPrivate = PrivateBrowsingUtils.isContentWindowPrivate(
    aProperties.source
  );
  download.source.windowRef = Cu.getWeakReference(aProperties.source);
This way the deserialisable in DownloadCore will just accept the source value as a string and store it in source.url (just as we need). We can then embellish it with the additional isPrivate and windowRef elements as in the code above.

In order to get this to work I also have to pull in PrivateBrowsingUtils at the top of the file: 
const { PrivateBrowsingUtils } = ChromeUtils.import(
  "resource://gre/modules/PrivateBrowsingUtils.jsm"
);
The end result is working. I've pushed all the changes to the remote repositories. Phew. Finally I'm going to call this task done.

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment
1 Jan 2024 : Day 125 #
It's 2024! Happy New Year to everyone. The start of the year is all about tidying up the PDF printing code for me, following the final steps to get things working yesterday. Since everything is now working, this is my chance to break things by making my changes cleaner, attempting to fix any edge cases and moving code around so there are more changes to Sailfish code and fewer changes to gecko-dev code.

Back on Day 99 when we first started looking at the problem I reverted an upstream change that removed a JavaScript class called DownloadPDFSaver. That is, the upstream change removed it, then when I reverted the change it put it back.

But it doesn't look like there's much in DownloadPDFSaver that's actually dependent on being inside gecko-dev, so it would make sense to move it into the embedlite-embedding JavaScript codebase instead. The embedlite-embedding code is specific to Sailfish OS and so entirely under our control. If I can move the changes there it will not only make things cleaner by avoiding a patch to the upstream code, it will also make the code much simpler to maintain for the future.

So my plan is to move everything related to DownloadPDFSaver into the EmbedliteDownloadManager.js file, which is the place where we actually need to make use of it.

So I've copied over the class prototype code with zero changes. I won't show it all here because there's quite a lot of it, but here's what the start of it looks like along with its docstring: 
/**
 * This DownloadSaver type creates a PDF file from the current document in a
 * given window, specified using the windowRef property of the DownloadSource
 * object associated with the download.
 *
 * In order to prevent the download from saving a different document than the one
 * originally loaded in the window, any attempt to restart the download will fail.
 *
 * Since this DownloadSaver type requires a live document as a source, it cannot
 * be persisted across sessions, unless the download already succeeded.
 */
DownloadPDFSaver.prototype = {
  __proto__: DownloadSaver.prototype,
[...]
This code essentially allows printing to happen but with a DownloadSaver interface which means we can hook it into our "downloads" code really easily.

Now previously we'd have called Downloads.createDownload() with the saver key set to "pdf". The original code that did this looked like this: 
            if (Services.ww.activeWindow) {
              (async function() {
                let list = await Downloads.getList(Downloads.ALL);
                let download = await Downloads.createDownload({
                  source: Services.ww.activeWindow,
                  target: data.to,
                  saver: "pdf",
                  contentType: "application/pdf"
                });
                download["saveAsPdf"] = true;
                download.start();
                list.add(download);
              })().then(null, Cu.reportError);
You can see the saver: "pdf" entry in there. This was consumed by the Downloads class from gecko-dev which handed it off to the DownloadCore class. However the upstream change carefully excised all of the functionality that handled this PDF code. All of the other download types were maintained, only the PDF code was removed.

So while previously I reverted the changes I've now created a new method inside EmbedliteDownloadManager.js in the aim of achieving the same thing and that looks like this: 
/**
 * Creates a new DownloadPDFSaver object, with its initial state derived from
 * the provided properties.
 *
 * @param aProperties
 *        Provides the initial properties for the newly created download.
 *        This matches the serializable representation of a Download object.
 *        Some of the most common properties in this object include:
 *        {
 *          source: An object providing a Ci.nsIDOMWindow interface.
 *          target: String containing the path of the target file.
 *        }
 *
 * @return The newly created DownloadPDFSaver object.
 */
DownloadPDFSaver.createDownload = async function(aProperties) {
  let download = await Downloads.createDownload({
    source: aProperties.source,
    target: aProperties.target,
    contentType: "application/pdf"
  });

  download.saver = new DownloadPDFSaver();
  download.saver.download = download;
  download["saveAsPdf"] = true;

  return download;
};
Note that DownloadPDFSaver is now in the same file, so can be used here just fine. This code will end up creating the structure that we talked about yesterday; the end result should look like this: 
download = Download
{
	source: {
	    url: Services.ww.activeWindow.location.href,
	    isPrivate: PrivateBrowsingUtils.isContentWindowPrivate(
	        Services.ww.activeWindow
	    ),
	    windowRef: Cu.getWeakReference(Services.ww.activeWindow),
	}
	target: {
	    path: data.to,
	}
	saver: DownloadPDFSaver(
	    download: download,
	),
	contentType: "application/pdf",
	saveAsPdf: true,
}
Some of the functionality is hidden inside the existing call to Downloads.createDownload() which creates a basic Download object that we then add our stuff to. If you look carefully you should be able to match up the code shown in our new DownloadPDFSaver.createDownload() method with the elements of the structure shown above.

With all this in place we can now change our code that creates the download like this: 
@@ -254,13 +269,10 @@ EmbedliteDownloadManager.prototype = {
             if (Services.ww.activeWindow) {
               (async function() {
                 let list = await Downloads.getList(Downloads.ALL);
-                let download = await Downloads.createDownload({
+                let download = await DownloadPDFSaver.createDownload({
                   source: Services.ww.activeWindow,
-                  target: data.to,
-                  saver: "pdf",
-                  contentType: "application/pdf"
+                  target: data.to
                 });
-                download["saveAsPdf"] = true;
                 download.start();
                 list.add(download);
               })().then(null, Cu.reportError);
To give our final version, which looks like this: 
          case "saveAsPdf":
            if (Services.ww.activeWindow) {
              (async function() {
                let list = await Downloads.getList(Downloads.ALL);
                let download = await DownloadPDFSaver.createDownload({
                  source: Services.ww.activeWindow,
                  target: data.to
                });
                download.start();
                list.add(download);
              })().then(null, Cu.reportError);
            } else {
              Logger.warn("No active window to print to pdf")
            }
            break;
Nice! But when I try to execute this code I hit several errors. Various classes and objects that were available in the DownloadCore.jsm file are no longer available here: DownloadSaver, DownloadError, OS and gPrintSettingsService.

Thankfully we can still pull these into our EmbedliteDownloadManager.js code to make use of them. I've added the following at the top of the file for this: 
const { DownloadSaver, DownloadError } = ChromeUtils.import(
  "resource://gre/modules/DownloadCore.jsm"
);

XPCOMUtils.defineLazyModuleGetters(this, {
  OS: "resource://gre/modules/osfile.jsm",
});

XPCOMUtils.defineLazyServiceGetter(
  this,
  "gPrintSettingsService",
  "@mozilla.org/gfx/printsettings-service;1",
  Ci.nsIPrintSettingsService
);
Now when I try to print everything works just as before. Lovely! It remains to revert the revert applied previously and to commit these changes to my working tree.

There is also now just one last task I need to do to round off this PDF print work and that's to "Record an issue for the setBrowserCover() hang." This is probably the easiest task of the lot, which is why I've left it until last. But that'll be a task for tomorrow.

Once this last step is done I can finally move on to something unrelated to the printing stack, which will be a relief!

If you'd like to read any of my other gecko diary entries, they're all available on my Gecko-dev Diary page.
Comment

Actions

Login Email Share