People Are Increasingly Choosing Private Web Search

DuckDuckGo has had a banner year:

And yet, DuckDuckGo. The privacy-oriented search engine netted more than 35 billion search queries in 2021, a 46.4% jump over 2020 (23.6 billion). That’s big. Even so, the company, which bills itself as the “Internet privacy company,” offering a search engine and other products designed to “empower you to seamlessly take control of your personal information online without any tradeoffs,” remains a rounding error compared to Google in search.

I use it. It’s not as a good a search engine as Google. Or, at least, Google often gets me what I want faster than DuckDuckGo does. To solve that, I use use the feature that allows me to use Google’s search engine through DuckDuckGo: prepend “!Google” to searches. Basically, DuckDuckGo launders my search.

EDITED TO ADD (1/12): I was wrong. DuckDuckGo does not provide privacy protections when searching using Google.

Tags: , , ,

Posted on January 6, 2022 at 6:29 AM • 59 Comments

Comments

tfb • January 6, 2022 6:57 AM

35E9 searches sounds like a lot. But perhaps I do 10 searches a day (I do more), so 3.5E3/y so that’s 10E6 active users if ‘active’ means 10 searches/day. They have a survey mentioned in the article which says 27E6 people in the US use it so if you guess 35E6 globally that means 3 searches/day each perhaps. Perhaps most people don’t search very much? Or perhaps people say they use it but don’t very much. Well, anyway, say about 30 million, so this is 1% of Facebook. Which isn’t that bad, actually.

(Note I’ve used DDG for a long time & I’m not disputing their numbers, I just wanted to work out what they mean as ‘searches/year’ is not a very helpful number I think.)

Q • January 6, 2022 7:00 AM

I don’t understand.

All it does for me is redirect my browser to google.com.

I could just go there by myself if I wanted to, without involving DDG in the process.

How does that make my search private? I still end up on the google.com doing the search there.

satai • January 6, 2022 7:08 AM

!g is enough for google search, you don’t need to type !Google. And you don’t need to prepend the search with it, you can use it in any part of the query.

gghgg • January 6, 2022 7:13 AM

You can use !g instead and it doesn’t have to prepend, it can be anywhere in the query…

Me, I use one of the google 3rd party frontends instead…

Charles Indelicato • January 6, 2022 7:14 AM

I get the intent of !Google. But this assumes I am using a browser where I am already identified to Google (i.e. Chrome), which means when the !Google hands off to Google, it’s no different from my searching in Google itself.

I use DDG browser for private searches (without logging into Google); I think if the concern is to hide search terms from G, it’s easier to do it in a separate browser entirely.

Dan • January 6, 2022 7:53 AM

You got this completely wrong – from DDG’s website “Bangs are shortcuts that quickly take you to search results on other sites… Remember, though, because your search is actually taking place on that other site, you are subject to that site’s policies, including its data collection practices.” Right at the top of the ‘bang’ page:

https://duckduckgo.com/bang

Winter • January 6, 2022 8:15 AM

@Dan
“Remember, though, because your search is actually taking place on that other site, you are subject to that site’s policies, including its data collection practices.”

You can search DuckDuckGo on their onion site in Tor:
ht-tps://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/

DuckDuckGo will have no data on you and neither will the endpoint search engine.

Greg • January 6, 2022 8:47 AM

!s will use Start Page instead, which does “launder” your google search.

DDG almost always works OK for me except for rare cases.

In those cases I retry with a !s prefix.

William Entriken • January 6, 2022 8:52 AM

For DDG users, simply add !g at the end or beginning.

For your muscle memory, if you are looking at DuckDuckGo search results and want to switch to Google for a sec, do this:

  • Command L
  • Right arrow key
  • Space-!-g
  • Enter

Richard Bejtlich • January 6, 2022 9:07 AM

It’s fascinating that one of the top people in security misunderstood the privacy implications of DDG’s “bang” feature. I hope we will have sympathy for the average user trying to navigate the complexity of protecting their security and privacy.

VanDiver • January 6, 2022 9:10 AM

Been using DDG for a long time now. My understanding had always been that DDG used Google to do “your” searches. The protection was that Google’s data collection would just point back to DDG and not you. Of course, once you click on a link your ISP and or DNS now know where you’re going, TOR and VPN’s aside.

anonymous • January 6, 2022 9:26 AM

I prefer Startpage, for privacy reasons, and have used it almost exclusively for years. Wasn’t happy about the sale of shares to that ad company, but at least Startpage is still directed and controlled by the original team, and I haven’t noticed anything sketchy yet.

I do use DDG occasionally, because sometimes Startpage throws up that stupid “Just checking…blah blah blah unusual activity…are you using a VPN….” message, and refuses to go any further. (Duh…of course I’m using a VPN; doesn’t everybody?) But I don’t really care for DDG much.

Clive Robinson • January 6, 2022 10:33 AM

@ ALL,

Of course, once you click on a link your ISP and or DNS now know where you’re going,

Is only part of the story…

For those using Google directly or not have you ever “moused over” those Google links to see what they realy are? Yup redirects through Google.

Have you ever checked where your DNS requests get sent? Yup Google or Cloudflare are most likely.

There’s a few other wrinkles as well.

My big problem with DDG is that it works fine with short search terms but goes nuts with long search terms. Oh and a few issues to do with the use of double quotes…

Many years ago I learned and became very adept at using complex search terms, once you get the hang of them they are very very hard to give up (like a fine wine, single malts, or those better Belgium Chocolates 😉

Marl • January 6, 2022 10:42 AM

There are lots of other privacy-oriented metasearch sites available. “Meta” meaning they don’t have their own crawlers; they use other crawlers like Google, Bing, etc.

Here’s a few:

SwissCows: https://swisscows.com/

MetaGer: https://metager.org/

Searx: https://searx.me/ (federated, can be locally hosted)

Mojeek: https://www.mojeek.com/ (has its own crawler)

Anyone that thinks StartPage is privacy-oriented is a little out of date. They were bought by System1 in 2019, and unless System1 is the first advertising company to acquire technology and refuse to use it to further its advertising ends… then this “does not compute.”

Gordon • January 6, 2022 11:09 AM

Sorry, but how can we be sure that DuckDuckGo won’t one day start selling our data? This takes only one crooked key person. One day they just decide that they need more money and hey, presto! – let’s sell, there’s companies that pay top dollar for that data that just sits here.

mark • January 6, 2022 11:30 AM

It’s not worse than google. Google, on the other hand (along with bing), if I go to “shopping”, it ignores any excludes. And I know it’s selling what I looked for to third parties, as well as using it itself.

Wannab techguy • January 6, 2022 11:36 AM

@Gordon,etal

That is my concern also. I’ve been using DDG for several years. It doesn’t take me to google(as far as I can tell). It’s set as my default search.
Lately, there have been tv & radio commercials from them. My first thought is, where are they getting the money from for those ads?

Arline • January 6, 2022 11:53 AM

Anyone that thinks StartPage is privacy-oriented is a little out of date. They were bought by System1 in 2019,

Ah, that explains why they’ve become so privacy-hostile in recent years. They often reject searches from Tor users, and while they claim to want feedback about this, they stopped accepting it via web form (I complained about this when it first started) and want it only via email which I have no anonymous way to use. Google, of course, has been blocking “robots” (i.e. anyone they can’t build a dossier on) for a decade or more, which mostly rules out using “!g” for anyone who wants privacy; it works maybe 5-10% of the time, and with poor results compared to its glory days.

DuckDuckGo supports searching via Tor—search “duckduckgo onion” there to get an official .onion address—but often gives bad results. I can’t count how many times I’ve put something in quotes and gotten pages that don’t include the phrase, sometimes don’t include one of the words at all. Bing always works, but not much better. Neither seems to support the “+” operator which Google used to, and don’t reliably support “-“. Sometimes they’ll return 0 results for some search that I know should have results, and then they’ll return results if I do the same thing with a site: operator (and I’m not talking about obscure pages—more like song lyrics easily found on fandom.com and elsewhere, for example). I’m reminded of Webcrawler, Hotbot, Lycos et al. in the late ’90s, and I’m waiting for a Google to come along.

If you want alternate (possibly privacy-respecting) front-ends for various web services, try this list: https://github.com/mendel5/alternative-front-ends
They’ve got some for Google. The Twitter frontends are useful too, since the official page explicitly refuses to display anything for people with Javascript disabled (sad for a site originally meant to display 140 characters of plain text).

schneier.com’s preview button still has no effect, so let’s hope for the best…

Arline • January 6, 2022 12:03 PM

@Gordon,

Sorry, but how can we be sure that DuckDuckGo won’t one day start selling our data?

Tor’s not perfect, but they do support searching via Tor and without cookies (not even an annoying cookie banner)—even via an official Onion service, as I mentioned in my previous message. I treat that as a good sign, and a bit of a canary. If they disable the onion service, or start blocking “robots” or wanting CAPTCHAs or cookies, be suspicious. It’s generally considered reasonably difficult for an Onion service operator to learn the identity of a Tor user, even if (as some people think) the intelligence agencies can break it. Onion services also preclude exit relays as an attack vector, since none are used; this also means the operator can’t even pretend to geolocate the users, hopefully precluding any geographic legal restrictions.

Matt • January 6, 2022 12:08 PM

@Gordon You can’t be 100% sure, obviously. The only way to be sure that a search engine isn’t collecting info about you would be to run your own search engine!

That said, you can never be 100% sure that anyone won’t turn on you at some point. Some level of trust is a fundamental requirement of interacting with other humans. If you decide that you don’t trust anyone enough to interact with them at all, well… that certainly is an approach to living life.

Winter • January 6, 2022 12:35 PM

“DuckDuckGo supports searching via Tor—search”

Qwant.com can be used from tor, albeit not onion, and is privacy friendly. It is reasonable good and has a usable interface. I believe it mainly uses Bing search.

I prefer it over DDG which somehow never returns results I can use.

lurker • January 6, 2022 1:22 PM

@wannabtechguy, where are they getting the money from for those ads?

If you search DDG for anything that can be bought or sold, the first one or two “results” are paid ad placements, and plainly so. I haven’t seen more than two, yet.

At least DDG has never offered me
“Find magnapinnidae on ebay” which you will get among the top three Ggl results looking for the subject of the previous squid.

lurker • January 6, 2022 5:14 PM

@Ted, All

I use qutebrowser on linux. The privacy mode must be good enough for Ggl. Every time I log in to a Ggl service with qutebrowser I get Security Warning email that somebody logged in to my account from a “new Linux device”.

Ted • January 6, 2022 5:54 PM

@lurker

Re: qutebrowser

Thanks for the heads up with that. It’s good to know there are options. Do you know how qutebrowser handles cookies? Also I wonder how linked these platforms are.

qutebrowser uses QtWebEngine by default. QtWebEngine is based on Google’s Chromium.

https://www.qutebrowser.org/doc/faq.html

JonKnowsNothing • January 6, 2022 9:39 PM

@Ted

re: What is it going to take for DDG to roll out a browser?

The money is not in the browser.

The money is in the search/sort/select databases on the back end.

DDG routes their searches through other engines, in theory stripping some identifiers from the query line. DDG leverages the back end databases already in existence.

iirc(badly) There have been some bumps along the way because the back end databases want all the data; not just some of it. There have been some dickering and horse trading in that regard.

It’s similar to the dickering and horse trading over news, news feeds and who gets to push the feed wagon.

Twitter banned news aggregator:

[Site] grew rapidly over the last twelve months by aggressively aggregating news stories published and reported by mainstream outlets.

Making a GUI isn’t nearly as interesting unless you have !COLOR TEMPLATES! as the primary attraction.

lurker • January 6, 2022 11:00 PM

@Ted, I’m not sure how qutebrowser’s longterm cookie management can be manipulated. I use it for my “private” browser; it holds session cookies during a session: close qutebrowser and cookies vanish.

There is a whole tribe of browsers “based on” Chromium. I don’t believe many of them hold any allegiance to Ggl. BTW qutebrowser’s default home page is the DDG search page, altho’ I prefer about:blank.

ResearcherZero • January 7, 2022 12:50 AM

Clive Robinson is right, unless your DNS queries are encrypted, then none of your requests are private. There are a number of other ways people are also tracked on the server or provisioning side.

Data is the new oil.
https://www.arnnet.com.au/article/611271/data-new-oil-especially-telcos/

Optus is part of SingTel and is a telco based in Australia that competes with the main telco and has about 35% of the market: cell phones, landlines, business, etc. Optus uses data mining in lots of ways: targeting communications, finding behavioral drivers, finding out what they want – understanding the customer.
hxxps://www.smartdatacollective.com/22044/

Any talk of data being anonymized should be viewed with skepticism.

“researchers from the University of Texas were able to re-identify individual Netflix subscribers by using a data set of 500,000 “anonymous” Netflix subscribers that had rated movies online, and matching it against a separate set of public data.”
hxxp://arxiv.org/PS_cache/cs/pdf/0610/0610105v2.pdf

“So what are all of the programs run by the N.S.A. or other federal agencies”
hxxps://www.nytimes.com/2013/11/15/us/cia-collecting-data-on-international-money-transfers-officials-say.html

Choose an option and apply it to a target.
hxxps://projects.propublica.org/nsa-grid/index.html

Don’t forget to remove your binary from the target…
hxxps://techcrunch.com/2016/08/16/everything-you-need-to-know-about-the-nsa-hack-but-were-afraid-to-google/

ResearcherZero • January 7, 2022 4:35 AM

Of course if companies which are given contracts to handle private data can’t get the basics of security right and design systems that are secure, and services they provide are not properly audited before they are handed the contract, then no one will have any privacy.

“The Cadence Health website only validated user input on the client side, not the server side, according to the hacker. This means that when a user accesses the tele-medicine site the normal way, by loading the site in their browser, they can only access their own data, but if they write a program that tries to access other data on the server, the server will respond with that data. The hacker simply asked the server for all patient data.”
https://theintercept.com/2021/09/28/covid-telehealth-hydroxychloroquine-ivermectin-hacked/

The Broward Health public health system has disclosed a large-scale data breach incident impacting 1,357,879 individuals.

Broward Health is a Florida-based healthcare system with over thirty locations offering a wide range of medical services and receives over 60,000 admissions per year.

An investigation revealed that the threat actors gained access to patient’s personal medical information, which may include the following items:

Full name
Date of birth
Physical address
Phone number
Financial or bank information
Social Security number
Insurance information and account number
Medical information and history
Condition, treatment, and diagnosis
Driver’s license number
Email address

https://www.bleepingcomputer.com/news/security/broward-health-discloses-data-breach-affecting-13-million-people/

Winter • January 7, 2022 4:51 AM

@ResearcherZero
“live Robinson is right, unless your DNS queries are encrypted, then none of your requests are private. There are a number of other ways people are also tracked on the server or provisioning side.”

That is not difficult to do. There are ample providers who offer this, e.g.:
ht-tps://dnscrypt.info/public-servers/

There are apps to set it up, e.g., on Android:
Nebulo
ht-tps://play.google.com/store/apps/details?id=com.frostnerd.smokescreen

Uthor • January 7, 2022 8:36 AM

@tfb, I’m sure there are days where I don’t do any searches at all. Though I’m a person who mostly goes to the same sites and finds things directly from there.

For the most part, I find DuckDuckGo just fine. Google is maybe a little better with breaking news, DuckDuckGo has a more pleasing image search (lets you view the full image directly instead of going to the original page), Google Maps is much better (which I use for local stores), DuckDuckGo doesn’t feed me into a shopping section.

Curious • January 7, 2022 9:53 AM

I’ve been using DuckDuckGo from early on I think. I am happily using it, and DDG seems to be adequate to my needs.

But, I can’t help but wonder if DDG is too good to be true. 😐

I trust nobody at all using my computer and I feel bad about using my computer most of the time. I wish I had a better impression of Linux, then I could keep using it. I am ofc ignorant about a lot of things, but I just wish things were better overall.

Grima Squeakersen • January 8, 2022 12:27 PM

@Clive re: “Many years ago I learned and became very adept at using complex search terms, once you get the hang of them they are very very hard to give up” – are you saying that you have the ability to do WWW searches using complex Boolean logic syntax arguments? One of the most disillusioning moments I have had on the web was when the search engine that best implemented that technique (Alta VIsta? Deja Vu?) went away (or stopped honoring that format, forget which). If that is the case, perhaps you could be so kind as to point to (or write 🙂 a guide on how to accomplish this. I, for one, would be most grateful.
-GS

Grima Squeakersen • January 8, 2022 1:16 PM

@marl et al re: DDG & alternatives – I use DDG as one of my search front-ends. I find it adequate, mostly. The biggest issue I have is that on some searches it seems to return quite a number of results that contain what appear to be strings of Cyrilic characters as well as English, and also appear to link to a type of web page that has been constructed more-or-less dynamically to match the search terms. There is probably some specific name for that practice, but I don’t know what it is. That leads me to wonder if DDG isn’t prioritizing (or at least failing to filter) results leading to scam/phishing sites to create for themselves an alternative revenue stream to directly marketing user data. If that is the case, I don’t see a tremendous difference in ethics or practical effects on privacy between that and the Google approach. I use Ecosia and Qwant as well. I always use a VPN and frequently change my apparent location by changing the VPN server, so it is unlikely that localization is playing any consistent role in the results I regularly see. I also have JavaScript disabled when I search, that was my main criteria for choosing those three front-ends. I have been intending to implement dnscrypt but hadn’t yet bit the bullet. These comments have motivated me to download the pieces and I will now proceed to implelent, so thanks for that!

Clive Robinson • January 8, 2022 2:31 PM

@ Grima Squeakersen,

If that is the case, perhaps you could be so kind as to point to (or write) a guide on how to accomplish this.

Unfortunately as you note Altervista is effectively no more and even Google is getting as dumb as a bucket of nails. You, I, and tens of thousands of librarians and other “information proffesionals” lament it’s passing and the dumbing down of systems.

The real reason it happened is “automation” a web crawler realy has no idea as to “content” and most web pages are not written to be indexed… So the result we have…

However as I’ve mentioned before, The Library of Congress is still as far as I am aware “the keeper of the candle of hope” for Z39.50,

https://en.wikipedia.org/wiki/Z39.50

Which was supposed to be the way professional information searching should go.

lurker • January 8, 2022 3:28 PM

@Clive: most web pages are not written to be indexed…

A few years back I had to disinherit a family member who was so proud of their new job, in Search Engine Optimization, which of course is nothing of the sort. I could not convince this person that poisoning a page by filling <meta> tags with handy dandy hit of the day words to grab the search engine regardless of content, was a crime against humanity.

SpaceLifeForm • January 8, 2022 4:51 PM

@ lurker, Clive

“just the facts ma’am, just the facts”

Crawlers should ignore most HTML tags.

It is too easy to inject junk into a webpage on the fly with PHP, and since the webserver can id the crawler, the webserver can poison a search engine on a selective basis.

Which is why you find a lot of junk.

Freezing_in_Brazil • January 8, 2022 5:04 PM

I hope this piece finds you all in excellent or improving health. May this year be a peaceful one for us all.

I use DDG too [rather dissatisfied with the insistent priority put on commercial links; the useful ones are often placed bellow sixth – but then it is “free”, I know…].

These days, I use Python libraries for a great deal of my web browsing and my own crawlers for lot of content aggregation/discovery pertaining to the universe of my interests. A lot of garbage is avoided this way [ads, common metadata and all that jazz]. If you don’t mind the effort and are data-inclined, you hardly need standard browsing software – or even search sites – anymore.

Best regards

Clive Robinson • January 8, 2022 5:49 PM

@ SpaceLifeForm, lurker, ALL,

Crawlers should ignore most HTML tags.

They dis and they didn’t depending on what bit of history you look at.

At one point web crawlers used to have the same “disinherit” attitude as @lurker. Anyone overloading a page would actually get “derated”.

But the abusers trying to get up the rakings would use other tricks. I guess an “evolutionary process” in effect started and it was not long before the search engines effectively stoped “derating” by page “over” content but by other means such as the number of independent pages linked in. But that became ruined by the online equivalent of “log rolling” where a group of web sites would cross link each other.

It’s acrually a very hard problem to resolve which is maybe why it feels like they have “given up”.

SpaceLifeForm • January 8, 2022 8:10 PM

@ Clive, lurker, ALL

In the olden daze, Google used links in a webpage as a constructive metric.

Early on (2 decades ago), it was actually a really good metric.

Not long after the link farms popped up.

Amie • January 8, 2022 9:49 PM

In the olden daze, Google used links in a webpage as a constructive metric.
Early on (2 decades ago), it was actually a really good metric.
Not long after the link farms popped up.

That’s an oversimplification. With PageRank, the value of a “page A to page B” link depends on the ranking of page A (and also decreases as the number of links on page A increases). In theory, that makes a link farm mostly worthless unless “important” pages link to it. So people will do things like post blog comments with links to the farm, or buy up reputable expired domains and replace them with farms, but these 2 specific examples would be relatively easy to guard against (e.g. by recognizing the HTML of popular blog software, and by tracking link appearance date vs. domain control changes). Links can still be a pretty good metric.

I just don’t get the impression that Google cares much about search spam. “Everyone” seems to know which sites are clogging up the results, and bitches about them—e.g., Quora now, ExpertsExchange in the past. One Google intern could knock out the worst offenders in a week and greatly improve results.

null clam • January 9, 2022 3:55 PM

Re: metrics

A couple of nice leisurely references:

Short:

How Google Finds Your Needle in the Web’s Haystack, David Austin, https://www.ams.org/publicoutreach/feature-column/fcarc-pagerank

More detailed:

Google’s PageRank and Beyond: The Science of Search Engine Rankings, Amy N. Langville and Carl D. Meyer, Princeton University Press, ISBN 9780691152660

To summarize (1) the “I”s have it ; (2) that’s the beauty of it, it does nothing.

SpaceLifeForm • January 10, 2022 8:05 PM

@ lurker, Winter, Clive

Note that the overbite plugin actually does do TLS, but just like dnscrypt, it is still going thru a proxy. So, there is still a MITM.

Note: gophers (secure) protocol does not exist.

Also, the plugin has a bug/feature if you do not load from about:blank as it appears to be quite capable to leak parent page URL.

SpaceLifeForm • January 13, 2022 8:46 PM

@ Bruce, ALL

EDITED TO ADD (1/12): I was wrong. DuckDuckGo does not provide privacy protections when searching using Google.

No search engine can. No one can.

That said, for privacy reasons, one could be better off, for one to follow various folk, that you learn that pay attention to specific subject matters, and let them distribute your search for you.

Learn who follows what subject matter.

Check them when something is happening.

You do not have to always check any specific search engine. Just follow subject matter experts, or at least those that pay attention.

Distribute your search.

Clive Robinson • January 13, 2022 9:11 PM

@ SpaceLifeForm, Bruce,

Just follow subject matter experts, or at least those that pay attention.

Or read what people write in watering holes such as this blogs Squid Page and similar.

The trick as always is seperating the wheat germ from the bran before milling 😉

JonKnowsNothing • January 13, 2022 11:42 PM

@SpaceLifeForm, @Clive, @All

re: Distribute your search & watch out for lions at Watering Holes

Axiom: Do Not Trigger The Red Alarm Bell Search

  • There are some Watering Holes that can land you: grounded forever.
  • There are some that will mark your checkbox with: the wrong answer.
  • There are some that can lead to: Vanished Status.
  • There are some that look: Very Interesting, until you get on the plane to FunLand and end up in NoLand.

For those in USA, be sure to fill out the US State Department Notice for: Authorized persons, to whom the USA can give my current address and whereabouts. A small form on the back pages, sometimes said to be No Longer Needed or Deprecated. Don’t leave home without it, and particularly in the possession of someone you trust to FindMe in NoT-Timbuktu.

Winter • January 14, 2022 12:33 AM

@ SpaceLifeForm, Bruce, Clive
“Just follow subject matter experts, or at least those that pay attention.”

Follow journalist rules:
1) One source is No source. Always have at least Two Independent sources.

2) An anonymous source is No Source. Look for someone whose name and background you can check.

3) Hearsay too is No Source. Go for the original witness.

If it involves anything scientific, it only counts if you find (and read) the peer reviewed publication. If there is no peer reviewed publication, it is just an anecdote. If it has not been reproduced by others, the result is tentative.

In this light, I hate it when newspapers do not link to the source of scientific news. When I find the research, it is not uncommon to see that the report does “not get it”.

Clive Robinson • January 14, 2022 7:26 AM

@ Winter, Bruce, SpaceLifeForm, ALL,

If it involves anything scientific, it only counts if you find (and read) the peer reviewed publication. If there is no peer reviewed publication, it is just an anecdote. If it has not been reproduced by others, the result is tentative.

The big problem with that is the academic community is both slow and very very patchy as well as playing “follow my leader”.

As can be seen from this blog, things discussed here take on average eight years to get published in a paper…

Then when you actually read the papers the authors are almost all young and it’s their first paper to a conference so is “novel” to atract attention of “best new paper committees”. They can do this as they have nothing to loose by not following a domain leader or current area of research.

To say academia in general has a “conservative outlook” is at best an understatment.

The research I do is frequently years ahead of what academia does, in some cases what I –and others– did back in the 1980’s is onky just appearing in academic papers. It’s not for want of knowledge, experience, or technology on academias part, in fact quite the opposit. It is the structure of academia that prevents them moving forward any faster than a gastropod on mogadon.

It’s why I use other measures to evaluate, one of which is,

1, Do the laws of physics as we currently know them alow this?

If yes then I look for what they say are the

2, Indicative signals of causation

Which is what you generally search for when coming up with a hypothesis. I then look carefully at,

3, The claimed related effect or outcome

And,

4, The claimed method from cause to effect.

Are they valid? Are there other effects or methods that are valid? If so can they effect the results and have they been eliminated by the experimental method used to test.

Does this stop me being fooled, no, but it would take a lot of work to come up with a fraud that would get by such reasoning. And yes whilst some people have gone to the effort of deliberatly doing such things and managed to fool a lot of people in the process, in general when you find a bad paper –and there are a lot out there– it boils down to the researcher fooling themselves for a whole heap of reasons.

Oh and never ever trust a paper where the test subjects have “free will”. The clasic example was “Rats in a maze” experiments. For years papers came out with some quite extrodinary claims about behaviour etc. Well on scientist was doubtfull so did his own research… What he found was that the behaviour scientists experiments were all fundemebtally flawed. That is the equipment used was not upto the standard needed to carry out the experiments. The mazes did not eliminate from the five senses, touch, sound or smell all of which could be used by the rats as methods of navigation. He then built a maze to eliminate those senses and found when carrying out previous tests the results the behavioural scientists had written about, nolonger showed above random expectation… He went on to show that the most likely sense the rats were using to navigate was the sound of their feet on the maze base board…

So added a new meaning to the old expression of,

“The pitter patter of tiny feet”

Bringing chaos into an orderly house.

Winter • January 14, 2022 8:52 AM

@Clive
“To say academia in general has a “conservative outlook” is at best an understatment.”

If you mean that academia wants to see evidence before it publishes, yes, indeed. Note that the better conferences nowadays do have peer review. The lead time for a conference paper is generally 1 year.

If you want quick results, you will have to accept quick disasters too. Peer review is often portrayed as a drag, but that ignores all the errors that are corrected in peer review. Whenever the peer review system fails, that means we get nonsense papers.

Also, the reason it is all the young people who write research papers is that they need to get a PhD and/or Post-Doc career. STEM papers are almost always collaborative efforts, sometimes with dozens of authors. In such a collaboration, the junior research are often assigned the first authorship as they need the citations most. If things go wrong, it is the last author(s) who are held responsible.

As for speed, you just have to look at the past two years in certain medical fields where results were published faster than anyone could read.

Clive Robinson • January 14, 2022 11:07 AM

@ Winter,

If you mean that academia wants to see evidence before it publishes, yes, indeed

No that is not what I am talking about, although that argument is often given as an excuse.

Academia is ferociously competative and your success in the research side is based on how much money you can pull in.

Part of being able to pull in money is to be a “leader in a field” where in some cases people will fall over themselves to offer you money.

How do you become a leader in a field well there are two basic ways,

1, Be the fields founder.
2, Publish lots of related papers.

The two are quite interrelated.

But the publishing of papers is a big problem…

Journals tend not to touch new fields of research. There are several reasons for this but it boils down to if you are a “big name” or not. If I write a paper and try to publish it, the chances are it won’t get published on the “who he?” determinant. If however I give it to our host @Bruce and invite him to put his name on then the chances of it being published go up enormously.

The side effect of this is “under my wing” behaviour, that is you do the research for a big name and they put their name on the paper and you get it published. Do enough of this and you slowely get a name of your own.

But the downside is big names have a reputation to preserve, therefore they become very conservative in their outlook and very narrow of focus.

It takes a very brave “big name” to “branch out” into a new field.

But a second issue, is if you are the founder of a new field of endevor who can be found to peer-review your papers?

It’s a chicken and egg situation.

I could go on but you can see the whole academic research side is actually designed to be against “new research” no matter what level you are at, just “incremental research”.

Which raises the obvious question of “How to change things?”…

As a side note, there is a museum to product design and market failure in Europe. It well documents the fact that in marketing well over 90% of investment in new products never shows a return to equalise the investment.

The actual return from investment in Scientific Research is hard to quantify as the returns tend to be predominantly “to society” not “to funders”. But we can clearly see just by looking around the room we are in or out the window that science without doubt has had a major and continuing repayment to just about everyone in society. So many orders of magnitude greater than the money “funders” provided to scientific researchers.

So perhaps the first way to answer the “How to change things?” question is “How to change the money flow?”. In the past governments have asked “big names” in Science and in “industry” in as authorative advisors. The results have always been bad, as basically they reflect what is best for the advisor not for either scientific research or society…

So as a second shot, maybe we should ask instead “How to change the way papers are published?”. Well the journals have such a bad name even “Nature” has realised it has to change. But there is so much money tied up in journals that change is at best nibbling around the edges which has caused all sorts of other issues.

One of the reasons I talk about the non proprietary research I do on this blog in general terms is to get past the nonsense that journals have become. Back if you like, to a time when science was at best young and researchers used to write to each other and discuse their ideas. Has it worked? Actually yes, I’ve been sited directly and the number of papers that can be easily seen to be about what I’ve discused is numbered in multiples of tens. Do I care that I don’t get credit, not realy as I’m not in that game. However I do care rather much about the effect on society, and I frequently wish there was a quabtifiable way of measuring it. Not for a monetary or similar reward[1] nor for status in other peoples eyes, no it’s just to be able to see that I’ve helped people even if they never get to know it. It’s one of the reasons I’m an “engineer” not a “scientist”[2] as I’ve indicated before.

[1] Though by me or our host @Bruce a drink even if it’s just a cup of tea would be nice, as I’ve said for quite a number of years now.

[2] Whilst scientists and engineers broadly overlap what they do, there is actually a difference in outlook. Scientists look for often abstract questions to answer, engineers look for answers to solve practical problems people have.

MrV • January 14, 2022 11:50 PM

I have completely moved all my searches over to DDG, and I am finding the results superior to google. This is particularly apparent if you use different computers that may have different google accounts/users logged in.
At least on DDG you search for something and get a list of results, compared to google where the results are skewed to the proclivities of a particular user.

AT • January 15, 2022 11:29 PM

What privacy guarantees can DuckDuckGo substantiate? Is their code open? Has it been audited by a third party? How do they make their money?

Any search engine can claim that they are private …

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.