Deprecated: Function create_function() is deprecated in /home/gladguys/public_html/demo/wordpress/ithare_updated/wp-content/plugins/categorized-tag-cloud/categorized-tag-cloud.php on line 220

Warning: Cannot modify header information - headers already sent by (output started at /home/gladguys/public_html/demo/wordpress/ithare_updated/wp-content/plugins/categorized-tag-cloud/categorized-tag-cloud.php:220) in /home/gladguys/public_html/demo/wordpress/ithare_updated/wp-includes/feed-rss2.php on line 8
IT Hare https://gladguys.com/demo/wordpress/ithare_updated on Soft.ware Thu, 01 Feb 2018 05:25:11 +0000 en-US hourly 1 https://wordpress.org/?v=4.9.12 Bot Fighting 201, part 3. ithare::obf: An Open Source Data+Source Randomized Obfuscation Library https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-3-ithareobf-an-open-source-datasource-randomized-obfuscation-library/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-3-ithareobf-an-open-source-datasource-randomized-obfuscation-library/#comments Tue, 09 Jan 2018 13:11:43 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Obfuscation: Growing Forest to Hide a Leaf
#DDMoG, Vol. VIII

[[This is Chapter 29(h) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

What a fiendish cipher it must be, to talk about it left, right, and center!

— from Cyberiad by Stanislaw Lem —

With all those considerations which we discussed over the course of last 2 posts, an obvious question of “how to use it in practice” arises. To illustrate those points I’ve made, I wrote a pure C++ template-based obfuscation library, and placed it to Github [NoBugs]. Note that it is a pure C++ solution, i.e. without an external code generator; while abilities of the pure C++ are limited compared to the code generator-based solutions, I was surprised how much it is possible to achieve while staying entirely within bounds of C++.1

DISCLAIMER: as of now, library is in pre-alpha stage; while I am reasonably confident that if it compiles, it will produce consistent results (i.e. output with obfuscation will be the same as output without obfuscation), chances of it failing to compile in certain use cases are still pretty high <sad-face />. Still, as a proof-of-concept it IS interesting to see the results which can be obtained even with a pure in-language solution.


1 C++17, to be exact; don’t even try it without MSVC2017, or without /std:c++latest. While support for modern Clang is hopefully possible, it is not there yet.

 

Basic Idea

As it was discussed above –

I am a Big Fan™ of automated randomized code generation.2

In other words – the idea is NOT to write obfuscations manually. Instead – we should just declare which-things-we-want-to-obfuscate, and then obfuscation library should take care of the rest.

Example: factorial() function

To illustrate “how ithare::obf can be used to obfuscate things”, we’ll use the following very simplistic factorial() function:

//Listing 29.factorial.orig
__declspec(noinline) int64_t factorial(int64_t x) {
  if(x<0)
    throw MyException(“Negative value of factorial!”);

  int64_t ret = 1;
  for(int64_t i=1; i <= x; ++i) {
    ret *= i;
  }

  return ret;
}

Note: above, __declspec(noinline) is used merely to isolate our factorial() function from the rest of code, to see what is going on more clearly. With a few very narrow exceptions, noinline SHOULD NOT be used for production code (leave alone obfuscated production code).

 

Baseline

After compiling the code above in Release x64 mode, we get the following assembly (as shown by IDA Pro; other disassemblers may provide less information, but we should be ready to the attacker using the-best-available-tools):

;;Listing 29.factorial.orig.asm
;;single-; indicates comments by IDA Pro
;;double-;; indicate my own comments
var_38 = byte ptr -38h;;offset identifier created by IDA Pro
sub rsp,58h
mov rdx,rcx
test rcx,rcx
js l1
mov eax,1
mov ecx,eax
cmp rax,rdx
jg l2
nop dword ptr[rax+rax+00000000h];;aligning instructions
l3: imul rax,rcx
inc rcx
cmp rcx,rdx
jle l3
l2: add rsp,58h
retn
l1: lea rcx,[rsp+58h+var_38]
call sub_2
lea rdx,[__TI1?AVMyException@@]; throw info
                               ;    for ‘class MyException’
lea rcx,[rsp+58h+var_38]
call _CxxThrowException

sub_2:
push rbx
sub rsp, 20h
lea rax,off_1400036F90
mov rbx, rcx
mov [rcx], rax
lea rdx, aNegativeArgume ; “Negative argument to factorial!”
add rcx, 8
call sub_3
mov rax, rbx
add rsp, 20h
pop rbx
retn

It is very short, and is rather clear even in asm. NB: as we can see, MyException name is visible in the .exe – that’s in spite of me disabling RTTI (RTTI seems to be enabled at least for classes-being-thrown regardless of /GR- option being fed to MSVC compiler <sad-face />).

Even better for the potential attacker (and worse for us), is that if we feed this code to the IDA Pro decompiler, our intentions will be even more obvious:

//Listing 29.factorial.obf.decompiled.c
signed __int64 __fastcall sub_1(signed __int64 a1)
{
  signed __int64  v1;//rdx
  signed __int64 result;//rax
  signed __int64 i;//rcx
  char v4;// [rsp+20h] [rbp-38h]

  v1 = a1;
  if(a1<0)
  {
    sub_2(&v4);
    CxxThrowException(&v4, &_TI1_AVMyException__);
  }

  result = 1i64;
  for( i = 1i64; i <= v1; ++i )
    result *= i;
  return result;
}

_QWORD *__fastcall sub_2(_QWORD *a1)
{
  _QWORD *v1;// rbx

  v1 = a1;
  *a1 = &off_140036F90;
  sub_3(a1 + 1, “Negative argument to factorial!”);
  return v1;
}

Aren’t you amazed at capabilities of IDA’s reverse engineering? I certainly am; it indeed looks extremely close to the original code, and our intentions are indeed very clear. Even if we wouldn’t give away that obvious clue with our “Negative argument…” error string – the attacker still won’t have much problems realizing that our sub_1() is a function which calculates factorial (that for(i) loop in sub_1() gives it away almost-instantly).

This is the point where our obfuscation task starts…

Obfuscated-but-Still-Readable-and-Debuggable?

As it was stated above, our goal is to obfuscate the binary code, while keeping our source code as-readable-as-possible. Let’s see how ithare::obf from library [NoBugs] deals with it.

To enable automated randomized obfuscation, we have to say which variables and which literals we want to obfuscate, and to what extent (leaving generation of the actual obfuscation code to the library and compiler). Let me show it on example of our factorial() function:

//Listing 29.factorial.obf
__declspec(noinline) OBF6(int64_t) factorial(OBF6(int64_t) x) {
  if(x<0)
    throw MyException(OBF5S(“Negative value of factorial!”));

  OBF3(int64_t) ret = 1;//we could obfuscate literal '1' too, 
                        //  but for such common literals it rarely makes sense
  for(OBF3(int64_t) i=1; i <= x; ++i) {
    ret *= i;
  }

  return ret;
}

That’s it! As you can see, we didn’t do any real obfuscation here, merely declaring which variables have to be obfuscated (in fact, we’re replacing types-to-be-obfuscated with their obfuscated counterparts).

About those digits within OBF6() and OBF3(). In ithare::obf there is a convention that this-digit-after-OBF means “how much we want to obfuscate this type/literal”: OBF0() means “obfuscate using no more than 1 CPU cycle”, OBF1() – “using no more than 3 CPU cycles”, OBF2() – “no more than 10 CPU cycles”, and so on. More generally, OBFX() means “obfuscate using no more than 10^(X/2) CPU cycles”, so OBF6() is allowed to eat up to 1000 CPU cycles.

Obfuscating Parameters and Return Values

In our example above, we obfuscated not only internal variables, but also input parameters and return values of our factorial() function (and moreover, we used MUCH more severe obfuscation on entry and on exit, than inside). The reason for it is that

We want to obfuscate not only the nature of the function, but also its interfaces.

This provides two very significant benefits:

  • When debugging, attacker cannot really see real values of the function parameters (he will see only those obfuscated values). As call stack is one of major tools used by attackers, making it less obvious (and we’ll be making it much less obvious – see decompiled examples below) certainly qualifies a Good Thing™.
  • Even if the attacker knows what our function does, to invoke the function properly he still needs to deobfuscate the API (i.e. reverse-engineer current randomized incarnation of the code). And as this obfuscation changes automagically on each build (more on it below) – it means that each successful-attack-which-uses-this-API, still has to be rewritten for each new release of our Client. Sounds as a lot of tedious mundane work for the attacker – and this is certainly another Good Thing™ for us as defenders.3

2 That is, for obfuscation purposes.
3 Of course, in theory this process can be automated, but for the time being – there are no such tools. Moreover – even in theory there are limits to such automation, especially as boundaries between real code and obfuscation are far from being obvious(!). Those reverse-engineering guys I spoke to, when faced with such automation, were referring to using SAT solvers for this task, and with SAT solvers being NP-complete – well, it is a pretty hopeless mechanism of dealing with sizeable problems.

 

Debugging

As we can observe, our source code is not too obfuscated. Moreover, if we compile it for debugging (i.e. without ITHARE_OBF_SEED being defined) – it will stay debuggable too; here is a screenshot from MSVC with our factorial() function being debugged:

debugging obfuscated code in Visual Studio

As we can see, in this mode (without ITHARE_OBF_SEED), all the variables look more or less as their counterparts (that added ‘val=’ to the value is not TOO bad debugging-wise). Of course, we won’t be debugging our obfuscation this way – but when debugging our app-level logic, we have to assume that whatever-obfuscation-we-have works flawlessly anyway.4


4 BTW, even now, in pre-alpha, I am reasonably confident that as soon as obfuscated code compiles – it does work properly (and bugs causing compile failures, are being steadily ironed out).

 

Compile for Production: Enter ITHARE_OBF_SEED

Now, let’s try to compile the very same Listing 29.factorial.obf for production; to do it – we have to recompile exactly the same source code but with ITHARE_OBF_SEED #defined to be a large random number.5

At this point, we get an executable, which is functionally equivalent to the original one – but is randomly and severely obfuscated. Let’s see what current (as of January 2018) version of the ithare::obf will do with our factorial() function.

From assembler, the following can be observed:

  • Our “Negative value of factorial!” literal is no longer easily visible (it doesn’t show in “Strings” view of IDA Pro, and is not really easily detectable until the moment we decide to use it).
  • Our simplistic factorial() function now takes whopping 2 kilobytes(!).

As it is sooooo large (and is not-so-easy-to-comprehend) – I will provide only a portion of it (specifically the inner loop) here:

;;Listing 29.factorial.obfuscated.asm.InnerLoopOnly
loc_140002430:
mov r9, r15
mov r10d, 1Fh
sub r9, rax
imul rax, rcx, 1Fh
mov r8d, r9d
sub r10, rax
and r8d, 0FFFF0000h
movzx eax, r9w
add r8d, eax
mov eax, r9d
sub r8, rax
add r8, r9
imul r10, r8
mov edx, r10d
movzx eax, r10w
and edx, 0FFFF0000h
add edx, eax
mov eax, r10d
sub rax, rdx
mov rdx, cs:qword_14003FAA0
sub rax, r10
add rax, r15
movzx r8d, byte ptr [rdx+2]
imul r8, -1Fh
imul rdx, rcx, -1Fh
add r8, rbx
mov rcx, r8
mov r8d, 1Fh
inc rdx
imul rcx, rdx
imul rdx, rcx, 1Fh
sub r8, rdx
cmp r8, r11
jle short loc_140002430

As we can see – it is far from being obvious; while we do have an imul operation within – it doesn’t give up much, as actually we have six of such imuls, and it isn’t obvious whether they’re part of obfuscation, or not.

And even if we look at the our obfuscated factorial() function through the prism of almighty IDA Pro’s decompiler – all we get is the following:

//Listing 29.factorial.obf.decompiled.c
_DWORD *__fastcall sub_1(_DWORD *a1, __int64 a2)
{
  int v2; // er14
  __int16 v3; // ax
  int v4; // er9
  int v5; // er11
  __int16 v6; // r10
  int v7; // er13
  int v8; // eax
  unsigned __int64 v9; // r8
  unsigned __int64 v10; // rax
  signed __int64 v11; // rcx
  int v12; // er8
  int v13; // er11
  int v14; // edi
  int v15; // edi
  int v16; // er11
  unsigned __int64 v17; // r11
  int v18; // er10
  unsigned int v19; // ebx
  int v20; // edi
  int v21; // er11
  unsigned __int64 v22; // rsi
  unsigned __int64 v23; // r10
  int v24; // er9
  unsigned int v25; // edi
  int v26; // ecx
  int v27; // er10
  int v28; // er9
  int v29; // er11
  unsigned __int64 v30; // rdx
  _DWORD *result; // rax
  unsigned __int64 v32; // rdx
  __int16 v33; // ax
  int v34; // er8
  __int64 v35; // r8
  _DWORD *v36; // [rsp+48h] [rbp-51h]
  char v37; // [rsp+50h] [rbp-49h]
  char v38; // [rsp+54h] [rbp-45h]
  char v39; // [rsp+58h] [rbp-41h]
  char v40; // [rsp+5Ch] [rbp-3Dh]
  char v41; // [rsp+60h] [rbp-39h]
  __int64 v42; // [rsp+70h] [rbp-29h]
  __int64 v43; // [rsp+78h] [rbp-21h]
  char v44; // [rsp+80h] [rbp-19h]
  unsigned int v45; // [rsp+A8h] [rbp+Fh]
  int v46; // [rsp+ACh] [rbp+13h]
  unsigned int v47; // [rsp+B0h] [rbp+17h]
  int v48; // [rsp+B4h] [rbp+1Bh]
  int v49; // [rsp+B8h] [rbp+1Fh]
  int v50; // [rsp+BCh] [rbp+23h]
  unsigned int v51; // [rsp+C0h] [rbp+27h]
  unsigned int v52; // [rsp+C4h] [rbp+2Bh]

  v36 = a1;
  v2 = -256 * BYTE5(a2) - ((unsigned __int16)(15 - HIWORD(a2)) << 16) - BYTE4(a2) + 31;
  v3 = (unsigned __int64)((unsigned int)(unsigned __int16)(HIWORD(v2)
                                                         - (-256 * BYTE5(a2) - BYTE4(a2) + 31)
                                                         * (-256 * BYTE5(a2) - BYTE4(a2) + 31))
                        - 15
                        + (v2 << 16)) >> 16;
  v4 = (((unsigned __int16)(HIWORD(v2) - (-256 * BYTE5(a2) - BYTE4(a2) + 31) * (-256 * BYTE5(a2) - BYTE4(a2) + 31))
       - 15
       + (v2 << 16)) << 16)
     + (unsigned __int16)(v3
                        - (HIWORD(v2) - (-256 * BYTE5(a2) - BYTE4(a2) + 31) * (-256 * BYTE5(a2) - BYTE4(a2) + 31) - 15)
                        * (HIWORD(v2) - (-256 * BYTE5(a2) - BYTE4(a2) + 31) * (-256 * BYTE5(a2) - BYTE4(a2) + 31) - 15));
  v5 = (unsigned __int16)((HIWORD(v4) << 8) + (((HIWORD(v4) >> 8) - HIWORD(v4)) & 0xFF))
     + ((((unsigned __int8)(3
                          - 15
                          * ((unsigned __int16)(v3
                                              - (HIWORD(v2)
                                               - (-256 * BYTE5(a2) - BYTE4(a2) + 31)
                                               * (-256 * BYTE5(a2) - BYTE4(a2) + 31)
                                               - 15)
                                              * (HIWORD(v2)
                                               - (-256 * BYTE5(a2) - BYTE4(a2) + 31)
                                               * (-256 * BYTE5(a2) - BYTE4(a2) + 31)
                                               - 15)) >> 8)) << 8)
       + 65521
       + (unsigned __int8)(v3
                         - (BYTE2(v2) - (31 - BYTE4(a2)) * (31 - BYTE4(a2)) - 15)
                         * (BYTE2(v2) - (31 - BYTE4(a2)) * (31 - BYTE4(a2)) - 15))) << 16);
  v6 = (unsigned __int16)(31 - ((unsigned __int8)-HIBYTE(v5) << 8) - (unsigned __int8)(3 - BYTE2(v5))) >> 8;
  v7 = ((_DWORD)a2 << 16) + (unsigned __int16)(WORD1(a2) - a2); v8 = (unsigned __int64)(v7 + (unsigned __int16)(WORD1(a2) - a2 - 3) - (unsigned int)(unsigned __int16)(WORD1(a2) - a2)) >> 16;
  v9 = 3i64
     - 3
     * (((v7 + (unsigned __int16)(WORD1(a2) - a2 - 3) - (unsigned __int16)(WORD1(a2) - a2)) << 16)
      + (unsigned int)(unsigned __int16)(v8 - (WORD1(a2) - a2 - 3) * (WORD1(a2) - a2 - 3)))
     - ((unsigned __int64)(3
                         * ((unsigned __int16)((unsigned __int8)(3 - v6)
                                             + (((unsigned __int8)(1 - (31 - (3 - BYTE2(v5)))) + ((31 - (_WORD)v5) << 8)) << 8))
                          - (unsigned __int16)(15
                                             - ((unsigned __int8)(3 - v6)
                                              + (((unsigned __int8)(1 - (31 - (3 - BYTE2(v5)))) + ((31 - (_WORD)v5) << 8)) << 8)))
                          - ((unsigned __int8)(3 - v6)
                           + (((unsigned __int8)(1 - (31 - (3 - BYTE2(v5))))
                             + ((unsigned int)(unsigned __int16)(31
                                                               - ((HIWORD(v4) << 8) + (((HIWORD(v4) >> 8) - HIWORD(v4)) & 0xFF))) << 8)) << 8))
                          + 5)) << 32);
  if ( (signed __int64)((v9 << 32)
                      + HIDWORD(v9)
                      - (3
                       - 3
                       * (((v7 + (unsigned __int16)(WORD1(a2) - a2 - 3) - (unsigned __int16)(WORD1(a2) - a2)) << 16)
                        + (unsigned __int16)(v8 - (WORD1(a2) - a2 - 3) * (WORD1(a2) - a2 - 3))))
                      * (3
                       - 3
                       * (((v7 + (unsigned __int16)(WORD1(a2) - a2 - 3) - (unsigned __int16)(WORD1(a2) - a2)) << 16)
                        + (unsigned int)(unsigned __int16)(v8 - (WORD1(a2) - a2 - 3) * (WORD1(a2) - a2 - 3))))) < 0 ) { v45 = 983040 * (((unsigned int)(3 * dword_14003DB80) >> 16) + ((unsigned __int16)(3 * dword_14003DB80) << 16)) + (unsigned __int16)((15 * (((unsigned int)(3 * dword_14003DB80) >> 16)
                             + ((unsigned __int16)(3 * dword_14003DB80) << 16)) >> 16)
                           - abs(15 * ((unsigned int)(3 * dword_14003DB80) >> 16)));
    v46 = 31
        * (((unsigned __int16)-(signed __int16)(((unsigned __int16)-(signed __int16)dword_14003DB84 >> 8)
                                              + ((unsigned __int8)-(char)dword_14003DB84 << 8)) << 16) + (unsigned __int16)-((-65536 * (unsigned __int16)(3 - HIWORD(dword_14003DB84)) - (unsigned int)(unsigned __int16)(((unsigned __int16)-(signed __int16)dword_14003DB84 >> 8)
                                                               + ((unsigned __int8)-(char)dword_14003DB84 << 8)) + 1) >> 16));
    v33 = (unsigned __int64)(unsigned int)-dword_14003DB88 >> 16;
    v47 = 15
        * (((-65536 * dword_14003DB88
           + (unsigned int)(unsigned __int16)(v33 - -(signed __int16)dword_14003DB88 * -(signed __int16)dword_14003DB88)) >> 16)
         + ((unsigned __int16)(-(signed __int16)dword_14003DB88 * -(signed __int16)dword_14003DB88 - v33) << 16));
    v34 = ((((unsigned __int16)-HIWORD(dword_14003DB8C) << 16)
          + (unsigned __int16)(15
                             * ((unsigned __int8)dword_14003DB8C + ((unsigned __int8)(1 - BYTE1(dword_14003DB8C)) << 8)))) << 16)
        + (unsigned __int16)(((((unsigned __int16)-HIWORD(dword_14003DB8C) << 16)
                             + (unsigned int)(unsigned __int16)(15
                                                              * ((unsigned __int8)dword_14003DB8C
                                                               + ((unsigned __int8)(1 - BYTE1(dword_14003DB8C)) << 8)))) >> 16)
                           - abs(
                               15
                             * ((unsigned __int8)dword_14003DB8C + ((unsigned __int8)(1 - BYTE1(dword_14003DB8C)) << 8))));
    v48 = (v34 << 16)
        + (unsigned __int16)(HIWORD(v34)
                           - abs(
                               ((((unsigned __int16)-HIWORD(dword_14003DB8C) << 16)
                               + (unsigned int)(unsigned __int16)(15
                                                                * ((unsigned __int8)dword_14003DB8C
                                                                 + ((unsigned __int8)(1 - BYTE1(dword_14003DB8C)) << 8)))) >> 16)
                             - abs(
                                 15
                               * ((unsigned __int8)dword_14003DB8C + ((unsigned __int8)(1 - BYTE1(dword_14003DB8C)) << 8))))); v49 = -65536 * (unsigned __int16)(4 - HIWORD(dword_14003DB90)) - (unsigned __int16)(((unsigned __int16)(45 - 15 * dword_14003DB90) >> 8)
                           + ((unsigned __int8)(45 - 15 * dword_14003DB90 - 15) << 8))
        + 1;
    v42 = 0i64;
    v43 = 15i64;
    v41 = 0;
    v35 = (((unsigned __int16)-HIWORD(dword_14003DB94) << 16)
         + (unsigned int)(unsigned __int16)(15
                                          * ((unsigned __int8)dword_14003DB94
                                           + ((unsigned __int8)(1 - BYTE1(dword_14003DB94)) << 8)))) << 16;
    v50 = v35
        + (unsigned __int16)(((((unsigned __int16)-HIWORD(dword_14003DB94) << 16)
                             + (unsigned int)(unsigned __int16)(15
                                                              * ((unsigned __int8)dword_14003DB94
                                                               + ((unsigned __int8)(1 - BYTE1(dword_14003DB94)) << 8)))) >> 16)
                           - abs(
                               15
                             * ((unsigned __int8)dword_14003DB94 + ((unsigned __int8)(1 - BYTE1(dword_14003DB94)) << 8)))); v51 = -15 * (unsigned __int16)qword_14003DB98 - 983040 * ((unsigned int)qword_14003DB98 >> 16);
    v52 = (HIDWORD(qword_14003DB98) & 0xFFFF0000) + (unsigned __int16)(45 - 15 * WORD2(qword_14003DB98));
    sub_2(
      &v41,
      31i64,
      v35,
      &v45,
      __PAIR__(dword_14003DB94, dword_14003DB8C),
      __PAIR__(dword_14003DB90, dword_14003DB84),
      qword_14003DB98);
    sub_3(&v44, &v41);
    CxxThrowException(&v44, &_TI1_AVMyException__);
  }
  v10 = 14i64;
  v11 = 930i64 * *(unsigned __int8 *)(qword_14003FAA0 + 2) + 1190112520884487202i64;
  v12 = (unsigned __int64)((unsigned int)(unsigned __int16)(HIWORD(v2) - v2 * v2) - 15 + (v2 << 16)) >> 16;
  v13 = (((unsigned __int16)(HIWORD(v2) - v2 * v2) - 15 + (v2 << 16)) << 16)
      + (unsigned __int16)(v12 - (HIWORD(v2) - v2 * v2 - 15) * (HIWORD(v2) - v2 * v2 - 15));
  v14 = (unsigned __int16)((HIWORD(v13) << 8) + (((HIWORD(v13) >> 8) - HIWORD(v13)) & 0xFF))
      + ((((unsigned __int8)(3
                           - 15
                           * ((unsigned __int16)(v12 - (HIWORD(v2) - v2 * v2 - 15) * (HIWORD(v2) - v2 * v2 - 15)) >> 8)) << 8)
        + 65521
        + (unsigned __int8)(v12 - (BYTE2(v2) - v2 * v2 - 15) * (BYTE2(v2) - v2 * v2 - 15))) << 16);
  v15 = (unsigned __int8)(3
                        - ((unsigned __int16)(31
                                            - ((unsigned __int8)-HIBYTE(v14) << 8) - (unsigned __int8)(3 - BYTE2(v14))) >> 8))
      + (((unsigned __int8)(1 - (31 - (3 - BYTE2(v14)))) + ((unsigned __int16)(31 - v14) << 8)) << 8);
  v16 = ((_DWORD)a2 << 16)
      + (unsigned __int16)(WORD1(a2) - a2)
      + (unsigned __int16)(WORD1(a2) - a2 - 3)
      - (unsigned __int16)(WORD1(a2) - a2);
  v17 = 3i64
      - 3 * ((v16 << 16) + (unsigned int)(unsigned __int16)(HIWORD(v16) - (WORD1(a2) - a2 - 3) * (WORD1(a2) - a2 - 3)))
      - ((unsigned __int64)(3 * ((unsigned __int16)v15 - (unsigned int)(unsigned __int16)(15 - v15) - v15 + 5)) << 32);
  if ( 31 - 31 * v11 <= (signed __int64)((v17 << 32) + (unsigned int)(HIDWORD(v17) - v17 * v17)) )
  {
    v18 = (unsigned __int16)(HIWORD(v2) - v2 * v2) + (v2 << 16) - 15;
    v19 = ((v18 << 16) + (unsigned int)(unsigned __int16)(HIWORD(v18) - v18 * v18)) >> 16;
    v20 = (unsigned __int16)(((_WORD)v19 << 8)
                           + ((((unsigned __int16)(((v18 << 16) + (unsigned int)(unsigned __int16)(HIWORD(v18) - v18 * v18)) >> 16) >> 8)
                             - v19) & 0xFF))
        + ((((unsigned __int8)(3 - 15 * ((unsigned __int16)(HIWORD(v18) - v18 * v18) >> 8)) << 8)
          + 65521
          + (unsigned __int8)(BYTE2(v18) - v18 * v18)) << 16);
    v21 = (unsigned __int8)(3
                          - ((unsigned __int16)(31
                                              - ((unsigned __int8)-HIBYTE(v20) << 8) - (unsigned __int8)(3 - BYTE2(v20))) >> 8))
        + (((unsigned __int8)(1 - (31 - (3 - BYTE2(v20))))
          + ((unsigned __int16)(31
                              - (((_WORD)v19 << 8)
                               + ((((unsigned __int16)(((v18 << 16) + (unsigned int)(unsigned __int16)(HIWORD(v18) - v18 * v18)) >> 16) >> 8)
                                 - v19) & 0xFF))) << 8)) << 8);
    do
    {
      v23 = (15
           - v10
           + (unsigned __int16)(15 - v10)
           + ((15 - (_DWORD)v10) & 0xFFFF0000)
           - (unsigned __int64)(unsigned int)(15 - v10))
          * (31 - 31 * v11);
      v10 = ((unsigned __int16)(15 - v10) + ((15 - (_DWORD)v10) & 0xFFFF0000)) * (31 - 31 * (_DWORD)v11)
          - (unsigned __int64)((unsigned __int16)v23 + ((unsigned int)v23 & 0xFFFF0000))
          - v23
          + 15;
      v11 = (-31 * v11 + 1) * (1190112520884487201i64 - 31i64 * *(unsigned __int8 *)(qword_14003FAA0 + 2));
      v22 = 3i64
          - 3
          * (((((_DWORD)a2 << 16)
             + (unsigned __int16)(WORD1(a2) - a2)
             + (unsigned __int16)(WORD1(a2) - a2 - 3)
             - (unsigned __int16)(WORD1(a2) - a2)) << 16)
           + (unsigned int)(unsigned __int16)(((((_DWORD)a2 << 16) + (unsigned __int16)(WORD1(a2) - a2) + (unsigned __int16)(WORD1(a2) - a2 - 3) - (unsigned int)(unsigned __int16)(WORD1(a2) - a2)) >> 16)
                                            - (WORD1(a2) - a2 - 3) * (WORD1(a2) - a2 - 3)))
          - ((unsigned __int64)(3 * ((unsigned __int16)v21 - (unsigned int)(unsigned __int16)(15 - v21) - v21 + 5)) << 32);
    }
    while ( 31 - 31 * v11 <= (signed __int64)((v22 << 32) + (unsigned int)(HIDWORD(v22) - v22 * v22)) ); } v24 = ((15 - v10 + (unsigned __int16)(15 - v10) + ((15 - (_DWORD)v10) & 0xFFFF0000) - (unsigned __int64)(unsigned int)(15 - v10)) >> 32)
      * (((unsigned __int16)(word_14003DB70 - 16916) << 16)
       + (unsigned __int16)(word_14003DB68
                          + 31518
                          + (unsigned __int8)(93 * (word_14003DB68 + 30) - 67 * *(_BYTE *)(qword_14003FAA0 + 2) - 74)
                          - (unsigned __int8)(word_14003DB68 + 30)));
  v25 = (((((unsigned __int16)(21587 - word_14003DB64 - 1) << 16) + (unsigned __int16)(-31 * (-31539 - word_14003DB60))) << 16)
       + (unsigned __int16)(((((unsigned __int16)(21587 - word_14003DB64 - 1) << 16) + (unsigned int)(unsigned __int16)(-31 * (-31539 - word_14003DB60))) >> 16)
                          - -31 * (-31539 - word_14003DB60) * -31 * (-31539 - word_14003DB60)))
      * ((unsigned __int16)(-21845 * HIWORD(v24))
       + ((unsigned __int16)((unsigned __int8)v24 - (unsigned __int8)(15 - v24) - v24) << 16));
  sub_4(&v38, &v37);
  v26 = 31 * ((unsigned __int64)sub_5(&v40, &v39) + 2032094506);
  v29 = ((unsigned __int16)(HIWORD(v26) - word_14003DB6C) + ((unsigned __int16)(31 * v26) << 16))
      * (v28 + ((v27 + (unsigned __int16)abs((_WORD)v28)) << 16));
  v30 = ((unsigned __int64)((v29 & 0xFFFF0000) + (unsigned __int16)v29) << 32) + (v25 >> 16)
      + ((v25 + (unsigned __int16)abs(HIWORD(v25))) << 16)
      + 31;
  result = v36;
  v32 = ((unsigned __int64)(-1431655765 * (_DWORD)v30
                          + (unsigned __int16)(21845 * v30)
                          - (unsigned __int16)(-21845 * v30)
                          + (unsigned __int16)(15 - 21845 * v30)
                          - (unsigned int)(unsigned __int16)(21845 * v30)) << 32)
      + (unsigned int)-((HIDWORD(v30) << 16) + 65537 * (HIDWORD(v30) >> 16));
  *v36 = HIDWORD(v32);
  v36[1] = ((v32 >> 32) + ((unsigned __int64)(unsigned int)v32 << 32)) >> 32;
  return result;
}

This is all what the-best-available-decompiler was able to infer from our automagically-obfuscated code

Not much if you ask me <wink />; figuring out what this function does – is very non-obvious to say the least (of course, it still does compute factorial, it is just that finding it out will take some hours – if not days). And the function-which-deals-with-our-exception (and which previously gave away the “Negative…” literal) now looks as follows:

//Listing 29.factorial.obf.sub_3.decompiled.c
_QWORD *__fastcall sub_3(_QWORD *a1, unsigned __int64 *a2)
{
  unsigned __int64 *v2; // rbx
  _QWORD *v3; // rdi
  unsigned __int64 v4; // rdx
  unsigned __int64 v5; // rcx
  unsigned __int64 v6; // rdx
  unsigned __int64 v7; // rax

  v2 = a2;
  v3 = a1;
  *a1 = &off_140037FC0;
  sub_6(a1 + 1);
  v4 = v2[3];
  if ( v4 >= 0x10 )
  {
    v5 = *v2;
    v6 = v4 + 1;
    if ( v6 >= 0x1000 )
    {
      if ( v6 + 39 <= v6 || v5 & 0x1F || (v7 = *(_QWORD *)(v5 - 8), v7 >= v5) || v5 - v7 - 8 > 0x1F )
        sub_7();
      v5 = *(_QWORD *)(v5 - 8);
    }
    sub_8(v5);
  }
  v2[2] = 0i64;
  v2[3] = 15i64;
  *(_BYTE *)v2 = 0;
  return v3;
}

We can see no literal in plain sight anymore – which is certainly a Good Thing(tm)…

In addition, none of the code in our factorial() function is relatively-easily-detectable “dead code”: all the code can be executed (and the code along those branches which don’t raise exceptions, is executed during normal operation of the program).

Bottom line:

While it is not 100% incomprehensible (nothing is), we did manage to obtain a very poorly readable machine code out of our rather-readable source.

This means that using this type of obfuscation, we’re creating an asymmetry in our favour – while we’re taking relatively limited efforts on our side, we’re increasing the attacker’s efforts by orders of magnitude. This kind of asymmetrical efforts is exactly what-good-protection should do <smile />.

Ever-Changing Binary Code

But wait – we’re still not done with the benefits of ithare::obf. On each subsequent release build of our Client, we should use different ITHARE_OBF_SEED.6

Using this technique, for each Client release, we’ll generate different (but still functionally equivalent) obfuscation code; this, in turn, will make a large chunk of attacker’s-efforts-coming-from-previous-build, useless.7

Let’s compare two decompiles of the exactly same source code, but with different ITHARE_OBFUSCATION_SEEDs (screenshot from https://www.diffnow.com/):

Difference between two obfuscations generated from exactly the same source code

As we can see, obfuscated code on both sides uses the same principles, but the code itself is very different; this, in turn, means that for each new Client build at least a significant portion of reverse engineering efforts should be repeated.

Moreover,

If we manage to release new versions of our Client faster than attacker can reverse engineer our automagically-generated stuff – bingo! We won!8

Once again, what we’re trying to do here – is to create an asymmetry, when something-which-comes-to-us-for-free (namely, automagically generated different obfuscation for each build) causes significant trouble to the attacker.


5 currently – only 64-bit seeds are supported, but in the future, I hope to extend seeds to 128-bit ones.
6 Ideally – we should use crypto-strength random number as our ITHARE_OBF_SEED.
7 Among other things, techniques such as F.L.I.R.T. and F.L.A.R.E. are rendered useless for the seriously-obfuscated functions.
8 Well, we’ll also have to integrate our obfuscations into our Client-Server protocol, which can be not-so-trivial (but still doable), more on it in [[TODO]] section below.

 

Caveats

Of course, as any tool of this kind, there are certainly caveats (in addition to any bugs-due-to-such-complicated-library-being-pre-alpha).

Still Security-by-Obscurity <sad-face />

First, most importantly, all-the-information-attacker-needs, is still in the code <sad-face />; what we’re doing, is merely hiding things, which cannot be really secure. This means that as-a-way-to-protect-your-intellectual-property (~=”your offline program from being pirated”), this method still isn’t likely to work9 (not to mention that motivation for most of such protections is controversial).

Still, in the context of MOGs, IMNSHO there is considerable value in hiding things in the manner described above (especially if such obfuscation is applied consistently across lots of the code). In particular, as we noted above, if attackers cannot cope with the pace of our new releases – it means that we won’t have problem with bot fighters/cheaters anymore. Sure, at some point they will invent tools-which-can-automagically-deobfuscate-our-stuff, but first, it will take time, and second – after they do it, we’ll be in position to readjust (enabling us to go into arms race with the attackers, which is already a tremendous improvement over the current state of affairs); moreover – my gut feeling is that in this battle we do have an upper hand (just because the attacker has to solve an inverse problem compared to us solving forward problem, and solving inverse problems tends to be much more complicated than solving forward problems).


9 OTOH, if speaking about programs which go online at least occasionally – this technique can be used to make hacks much more difficult (how – is beyond the scope of this book, but a requirement for a program to go online will allow…)

 

Performance

As for performance – of course, convoluted obfuscated binary code does take its toll; that’s exactly why all the obfuscation primitives are supposed to mention maximum-number-of-CPU-cycles-we-can-spare. And of course, for 3D vertex-level code using any obfuscation is a fallacy.

On the other hand – it is well-known that 5% of code take 95% of CPU time (and conversely, 95% of code take only 5% of CPU time); this tends to stand for games too (in particular, script engines tend to be much less optimized than 3D graphics). As a result, our solution is rather obvious: let’s obfuscate non-performance-critical code.

Moreover,

It is non-performance-critical-code which tends to be of the most interest to the attackers(!)

Indeed, trying to extract usable information (such as “opponent is at 11 o’clock”) from vertex-level manipulations is a gargantuan task; it is usually much easier to get the opponent’s coordinates right after the network-packet-with-these-coordinates is parsed; however – as this parsing doesn’t happen too often, it can be obfuscated without causing too much of the performance penalty.

Let’s make a quick calculation. Our non-obfuscated factorial() function (when called to calculate factorial(20)), takes about 10ns of time. Our seriously-obfuscated version shown above, takes about 200ns – which is whopping 20x longer. However, let’s not be hasty to write it off on this “20x slower” basis. If we’re using some-function-obfuscated-in-a-similar-manner to parse our network packets – we’ll have to call it only once per network tick, i.e. 20 times per second; under these conditions – adding 200ns per network tick will take only 4 microseconds out of each second, or mere 0.0004%. In other words –

we can easily use as many as 10000 such-obfuscations-as-factorial()-above-uses per each network packet – and performance penalty will still be unnoticeable.

Sure, we DO need to be very careful about performance – but the division between “high-performance code” and “code in need for obfuscation” is there, so it is possible to exploit it.

Open Source Obfuscation is NOT an Oxymoron

At the first glance, saying “open source” and “obfuscation” in the same breath sounds as an oxymoron; how can we possibly both obfuscate things and make them 100% public at the same time? However, as soon as we say that we’re using randomized code generation – it starts to make sense.

While ithare::obf itself is open-source (and therefore can be validated by lots of people(!)), ITHARE_OBF_SEED parameter which was used to compile each of the builds, is certainly intended to stay out of sight, so that the attacker won’t know what exact obfuscation was generated for the executable he’s dealing with. This, in turn, means that

It IS possible to have ALL the code open, just ITHARE_OBF_SEED private – and still enjoy some protection provided by obfuscation

In a sense – it is similar to having encryption algorithms completely public, with just their keys being secret; here – ithare::obf plus our obfuscated-source-code is our public algorithm, and ITHARE_OBF_SEED is our key.

In particular (as we’ll discuss in [[TODO]] section below) we can try to use similar obfuscation to obfuscate Client-2-Server protocols in a 100%-open-source game (just not publishing specific ITHARE_OBF_SEED used for building a specific Server+specific Clients). This, in turn, would allow anybody to compile and run their own Server+Client; it is just that ITHARE_OBF_SEED will be unknown for each such Server+Client pair – so while all such Servers+Client pairs are functionally the same (and play the same game) – protocols for each such pair will be different, with the automagically-generated obfuscation code providing non-trivial protection from cheaters and bots (and moreover, for each such Server+Client pair obfuscation will need to be broken separately).

In addition, even now ithare::obf has a modular structure, where it is rather easy to add your own primitives (and ithare::obf-with-your-own-primitives is no longer 100% open source, which may both provide additional protection and give you more peace of mind). In the future, I expect to provide an “official” way to expand ithare::obf with your-own-primitives without the need to modify its source code.

Summary on ithare::obf

To summarize our findings so far:

  • It is possible to have a code which is both rather readable in source, and pretty much unreadable in binary
    • As a side benefit – it is possible to re-generate our binary code on each build at virtually zero cost for us, (while making the life of attackers much more complicated)
  • ithare::obf library is currently immature (and certainly NOT production-ready), but it does provide proof-of-concept implementation for these things
    • Even current state-of-the-art IDA decompiler is no match for even-current rather-basic implementation of ithare::obf
  • It is still not a silver bullet – but can be used as a building block to build very serious protection from cheaters (in fact, most of anti-cheater protection techniques would benefit from this kind of randomized obfuscation code).
  • Performance-wise, we can do A LOT of obfuscation per network tick
    • And apparently, it is rather-performance-non-critical stuff which is the easiest to attack, so it is exactly the thing which we should obfuscate (alongside with any anti-cheating routines we want to have)
  • Open-source obfuscation is possible (as long as we keep seed out of sight); in a sense – it is about the same as having crypto-algorithm public, while keeping its key secret (with our ithare::obf directly corresponding to the crypto-algorithm, and our seed corresponding to the key).

[[To Be Continued…

This concludes beta Chapter 29(h) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(i), where we’ll discuss issues related to protocol obfuscation (including versioning)]]

 


References

[NoBugs] 'No Bugs' Hare, C++ Code+Data Obfuscation Library with Build-Time Polymorphism

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-3-ithareobf-an-open-source-datasource-randomized-obfuscation-library/feed/ 2
Bot Fighting 201, Part 2: Obfuscating Literals https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-2-obfuscating-literals/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-2-obfuscating-literals/#respond Tue, 02 Jan 2018 14:33:58 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Obfuscating Literals
#DDMoG, Vol. VIII

[[This is Chapter 29(g) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

In previous instalment, we discussed a few basic techniques related to Data+Code Obfuscation; most of the time, we were speaking about bijections.

On the other hand, bijections taken alone are NOT exactly sufficient to obfuscate our code; this phenomenon is related to two observations:

  • Literals can tell a LOT about our program.
    • Moreover, if using literals in our bijections – they can easily give away the correspondence between our obfuscation and deobfuscation routines – and this is one of those pieces of information which are better to be kept out of plain view.
  • If trying to obfuscate literals using simple bijections-over-literals – first, our own compiler will try to optimize constants out, and then attacker’s decompiler will try to optimize out the rest.
    • In other words – if using simple bijections-over-literals, we cannot be sure that our supposedly-obfuscated literal looks reasonably-obfuscated to the hacker, or not <sad-face />.

As a result, we DO need to find a somewhat-special way to obfuscate literals (to make sure that compiler+decompiler doesn’t optimize it out too easily). In this regard, several different approaches are possible.

Volatile

One very simple1 way of obfuscating literals is to generate a random constant CC (just as above, at build-time), and then to generate the following code:

template<>
struct obfuscated_literal<size_t,13,456> {
  //13 is the constant we’re hiding.
  //456 is an ID of this obfuscation (similar to
  // the obf<> discussed above)
  static size_t volatile c;
  static constexpr size_t CX = obfuscate(13+CC);
  //obfuscate() should be constexpr function,
  // not a big deal in C++17

  static size_t value() const {
    return deobfuscate(c)-CC;
  }
};
volatile size_t obfuscated_literal<size_t,13,456>::c = CX;

As variable c is declared as volatile (~=”allowed to be changed at any moment without any reason-known-to-the-compiler”), the compiler just doesn’t have any options other than to read it (and on each and every access to it too); this ensures that the call to deobfuscate() won’t be optimized out.

A variation of this technique is related to having a table-based implementation of the deobfuscation, where the table-used-for-deobfuscation, is declared volatile.


1 and supposedly pretty much bulletproof at least against our own compiler optimizing it out

 

No-inline Functions

In addition to volatiles, I had good experience with using a non-inlineable (__declspec__(noinline)) function calls to hide constants. Even better – I made them use potential-pointer-aliasing (which is a big headache for compiler writers, effectively prohibiting them to make certain assumptions, good for us <wink />!):

__declspec__(noinline)
size_t aliased_pointer_func(size_t* x, size_t* y) {
  *x = 0;
  *y = 1;
  return *x;//can be either 0, or 1,
            // depending on
            // pointers x and y being equal or not
}

TBH, I don’t really see how a call to such a function can be optimized out (unless compiler ignores “noinline” specification); apparently, at least those compilers I tried, agreed with me on it, and left the call in place <smile />.

Reading OS memory

Elaborating a bit further on the idea of obfuscating things (and at the same time, doing poor-man’s anti-debugging for free <wink />) via reading-some-memory, we can write something along the following lines:

template<>
struct obfuscated_literal<size_t,13,457> {
  static constexpr size_t CX = obfuscate(13);
  static size_t value() const {
    uint8_t* peb = (uint8_t*)__readfsdword(0x30);
    uint8_t beingDebugged = peb[2];
    return deobfuscate(CX*(beingDebugged+1));
  }
};
//Warning: at some point, I run into a code generation bug with current MSVC compiler
// related to __readfsdword() intrinsic; as a result, I do NOT recommend to use 
// __readfsdword() in generated code; for a seemingly-working example of using it in a safe 
// manner, see [[TODO]]

The idea here is actually rather neat: if beingDebugged is (as we expect) 0, then beingDebugged+1 is 1, and then value() will return 13. However, if we’re run under debugger, the result will be very different, leading to crash (or to pretty-much-endless-loop) very soon (and without letting hacker know what exactly went wrong). Moreover, this technique is NOT limited to reading specifically BeingDebugged flag; in fact – most of the in-memory flags discussed above (including heap flags, NTGlobalFlag, etc.) can be used in this manner.

Unfortunately, all such memory locations are well-known and are routinely patched by ScyllaHide. It means that this technique won’t be able to seriously help our defense efforts – but as it comes pretty much for free (while solving our task-at-hand=”making sure compiler doesn’t optimize literal obfuscation out”) – why not use it?

Changing Variables with Invariants

All the methods above are working – but on the other hand, each of them can be detected relatively easily; while as of now, I don’t know of tools doing it, I am pretty sure that they can (and probably will) be developed as the time passes. However, there is one further thing which we can do to complicate such analysis (and IMO, very significantly too).

For example, we could use the following code to obfuscate our literal 13:

template<>
struct obfuscated_literal<size_t,13,458> {
  static size_t c = 0x3475723D;//note that 0xD == 13
  static constexpr CC = some_random_constant * 16;
  static size_t value() const {
    c += CC;
    return c & 0x0F;
  }
};

The point here is that while the variable c is changing on each call to value() (so it cannot be detected as a ‘de-facto constant memory location’), the last 4 bits of c are always 0xD, so value() function always returns 13 (as it should).

Moreover, this is certainly not the only technique which allows us to have ever-changing variable which does keep certain invariant (which we can rely on to derive our constant from). In particular, at least the following ever-changing-variables-with-invariants are possible:

  • Generalization of the technique above to any number of bits.2
  • If we realize that actually, in the code above &0x0F is equivalent to mod 16 (and ‘+=’ is actually addition mod 2^32 where 2^32 is a multiple of 16), we can generalize the technique above to operations mod M1 and mod M1*M2, where neither M1 nor M2 is 2^n.3
  • Actually, any finite cyclic subgroup will do here. Unfortunately, I have to admit that I am extremely incompetent in group theory, however:
    • it is perfectly possible to generate such a subgroup defined in a table manner (and if we define it over uint8_t, implementation will be very practical too).
  • Also, XOR-ing our variable with a constant will ensure that only some of the bits will change, and all the others remain intact (with those intact bits effectively forming our invariant). This, in turn, can be generalized to the following:
    • Considering our variable as a bunch of ‘digits’.
      • To confuse things better, it is better to use non-binary ‘digits’.
        • To confuse things even further, valid range for each ‘digit’ can be made different.
    • On each iteration, change only some of the ‘digits’ (say, in a pseudo-random manner), keeping the rest of the ‘digits’ intact.
    • Those intact digits form our invariant, and can be used to calculate our literal.

2 as long as it is strictly less than variable size
3 Beware: with M!=2^n, we MUST avoid overflow mod 2^32. For a supposedly-working example, see [[TODO]]

 

Changes and Threads

From the practical perspective, using ever-changing-variables-with-invariants requires being very careful in presence of multithreading <sad-face />. In particular:

  • Formally, unless we’re using thread_local variables,4 we have to ensure that our access to the variable is atomic.
    • Note that for our purposes, there is no need to ensure atomicity of the whole read-modify-write operation; instead, it is sufficient to have both read and write as atomic (and if there is an intervening write coming from a different thread – it cannot violate invariant anyway).
    • In practice – as of now, even simple reads/writes to properly-aligned-variable will do, but formally doing so qualifies as a dreaded Undefined Behavior, so nobody knows when it starts hurting us <sad-face />
    • OTOH, std::atomic<> is formally correct (and without too much overhead too)
  • More importantly, from the practical point of view, we have to ensure that we don’t thrash CPU caches by modifying our ever-changing variable from different threads too often. From my current perspective, two approaches are possible in this regard:
    • Say that the potential cost of this trick is huuuuge (like 100+ cycles), so we won’t use it in all-but-the-most-non-performance-critical obfuscations.
    • Use ever-changing-variables only with thread_local specifier, which by definition eliminates all the MT-related problems (at the cost of having storage for these variables for all our threads, but this is rarely a problem, at least as long as we keep number of our generated thread_local variables to some hundreds).

 

Obfuscating String Literals

Up to now, we were discussing obfuscating integer literals; however – in practice it is even more important to obfuscate string literals. One traditional way of obfuscating string literals is to have strings-stored-in-data-segment XOR-ed with a single byte; it is a very-well known technique – and is easily defeated too <sad-face />; in particular – such a single deobfuscation routine can be easily found and hacked, revealing all the supposedly-obfuscated strings to the attacker.

However, we can (and SHOULD) do MUCH better than that. Indeed, if we apply our approach discussed-above-with-regards-to-integer-literals, to string literals – we’ll be able to have orders-of-magnitude better protection than naïve XOR-with-byte. In particular, if we (a) generate individual obfuscation code for each of the string literals, and (b) have generated obfuscation of the same length as literal-being-obfuscated – we’ll get the following very-substantial benefits:

  • There WON’T be one single point of attack.
  • Each and every obfuscation has to be hacked individually.
    • Moreover, if we re-generate our obfuscation code on every build – it has to be done from scratch for each new build.
  • Deobfuscation won’t contain XORs (which are screaming out loud “we’re deobfuscating some stuff here!”
  • If we like it, deobfuscation can be made loop-less.

4 or otherwise guaranteeing that access to the same ever-changing-variable from different threads is impossible.

 

[[To Be Continued…

This concludes beta Chapter 29(g) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(h), where we’ll discuss my open-source C++17 obfuscation library – which does most of the things discussed in “Bot Fighting 201” more or less for free]]


Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-part-2-obfuscating-literals/feed/ 0
Bot Fighting 201: Declarative Data+Code Obfuscation with Build-Time Polymorphism in C++ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-declarative-datacode-obfuscation-with-build-time-polymorphism-in-c/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-declarative-datacode-obfuscation-with-build-time-polymorphism-in-c/#comments Tue, 26 Dec 2017 23:03:54 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Obfuscation: What You See Is NOT What You Get
#DDMoG, Vol. VIII

[[This is Chapter 29(f) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

After we discussed Bot Fighting 101103 (which were about techniques which are commonly used, but are mostly inefficient against serious hackers), we can proceed to a much more interesting topic: how to increase resilience of our Client-Side program to those Brute-Force Attacks discussed in [[TODO]] section above. To re-iterate – Brute-Force Attacks tend to start with identifying system calls (which are pretty much impossible to hide completely) – in the context of MOG it is usually a socket call; after a system call is identified – the attacker goes up the call stack to get to the-place-of-interest-to-him (usually – starting from that-point-where-the-message-to-be-sent-over-the-socket-is-prepared, and going up from there).

At this point, we’ll discuss one serious anti-hacking measure, which is aimed to:

  • Keep our source code perfectly readable (and with minimal changes);
  • Make our binary code as-unreadable-as-possible
  • Obfuscate in-memory data
  • Make sure that even if source code stays the same – binary code changes a lot on each build. In other words – we’re aiming to have build-time code polymorphism.

It may sound too-good-to-be-true, but apparently, it is perfectly achievable at least in C++; let’s take a closer look at this technique.

Preliminaries – How Our Source Code Will Look

To illustrate our approach, let’s take an extremely simplistic function which we want to obfuscate:

//Listing 29.factorial.orig
uint32_t factorial(uint32_t x) {
  uint32_t ret = 1;
  for(size_t i=1; i <= x ; ++i) {
    ret *= i;
  }
  return ret;
}

NB: in our exercises, we’ll be using MSVC 2017 (in Release mode) – as it is probably the most popular compiler for the Client-Side; for other compilers, the results will be similar, though certainly not identical.

NB2: for my tests, I used __declspec(noinline) for factorial() to make sure that compiler doesn’t inline it (which would complicate our analysis); it is a TEST-ONLY feature, and as a Bit Fat Rule of Thumb™, in production code we should be very aggressive with inlining, and avoid any __declspec(noinline).

When we compile our factorial() function, we get the following asm:1

;Listing 29.factorial.orig.asm
push esi
mov esi,1
mov edx,esi
cmp ecx,esi
jbe l2
nop dword ptr[eax]; aligning next instruction
l1: imul esi,edx; ret *= i
inc edx; ++i
cmp edx,ecx
jb l1
l2: mov eax,esi
pop esi
ret

Here, in spite of all the attempts of the compiler to make things less clear, there is an obvious loop starting from label l1, with a multiplication inside. As the result of the multiplication is multiplied again and again with an ever-incremented counter – it is relatively easy to guess that we’re dealing with calculating factorial here.

Now, let’s see what we can do to make the code less obvious to the attacker, while preserving clarity of our source code. Let’s introduce an obf<> template, defined as follows:

//Listing 29.obf
template<class T, int ID>
//Here, ID is an ‘identifier’ of
//  the instance of our obfuscation
//  We DO want to keep them different, don’t we?
class obf {
  T val;

  public:
  obf() {}
  template<class T2>
  obf(T2 v) { val = v; }
  T value() const { return val; }
  operator T() const { return value(); }

  template<class T2>
  obf& operator *=(T2 t) { val *= t; return *this; }
  //NB: in this context, I tend to prefer templatized member functions
  //  to free-standing friends, because it is generally easier
  //  to address issues from improper ordering, than from type
  //  mismatches. And having templatized free-standing friends
  //  is not an option due to ambiguities. OTOH, if you prefer
  //  free-standing non-templatized friends – they’re ok too.

  //do the same for =, +=, -=, /=, %=
  // and IF you need them – for bitwise ops too

  obf& operator ++() { val++; return *this; }
  obf operator ++(int) { return obf(val++); }
  //do the same for –-

  template<class T2>
  bool operator <(T2 b) { return val < b; }
  //do the same for +, -, *, /, %
  // and IF you need them – for bitwise ops too
};

Now, we can rewrite our factorial() function into the following:

//Listing 29.factorial.obf
obf<uint32_t,123> factorial(obf<uint32_t,124> x) {
  obf<uint32_t,125> ret = 1;
  for(obf<size_t,126> i=1; i < x ; ++i) {
    ret *= i;
  }
  return ret;
}
//BTW, we’re NOT required to have ALL the variables
//  obfuscated; in fact – it should be done on case-by-case
//  basis

NB: let’s keep in mind that while obf<uint32_t> is very close to uint32_t, it is not a ‘perfect wrapper’ (in particular, because of ‘1 built-in + 1 user-defined’ rule for implicit conversions). It means that some changes MIGHT be necessary to the code after we replace type with obf<type> in some declaration – but they’re usually solvable very easily by replacing variable with variable.value(). Also – make sure NOT to add any casts when enabling obf<> types; adding casts opens a huuuuge can of worms able to introduce LOTS of subtle bugs; fortunately – from what I’ve seen, 99% of the code works after replacing type with obf<type> ‘as is’, and adding .value() allows to deal with the rest.

As we didn’t really change anything, this code still produces the same results when we run it; apparently, asm did change a bit, but the inner loop (the one which reveals all the stuff to the attacker), is still exactly the same.2

Now, we can go a bit further and avoid writing those IDs manually, making an OBF() macro to wrap our obf<> type:3

#define OBF(t) obf<t,__LINE__>
//If you feel like it and use C++17,
//  you can even use constexpr function of (__FILE__,__LINE__)
//  as an ID; in C++14 it may be possible
//  but is hardly worth the trouble

Then, our factorial() function will start to look as

//Listing 29.obf.macro
OBF(uint32_t) factorial(OBF(uint32_t) x) {
  OBF(uint32_t) ret = 1;
  for(OBF(size_t) i=1; i < x ; ++i) {
    ret *= i;
  }
  return ret;
}
//Again, we’re NOT required to have ALL the variables
//  obfuscated

IMO, it looks reasonably close to the original; moreover, I’d argue that it is probably the best thing we can have in this regard. In general, we DO need to specify which variables/parameters have to be obfuscated, and moreover – have to specify how performance-critical they are. In practice, I usually suggest to use template classes such as obf0<>, obf2<>,… obf5<> etc. (with ‘obf0<>’ meaning ‘obfuscate lightly’, and ‘obf5<>’ meaning ‘obfuscate heavily’), and corresponding macros OBF(), OBF2(),…, OBF5().

By now, we’re done with the changes to our source code;4 now, let’s see how these (IMO rather minor) changes allow to make the life of the hacker by orders of magnitude more difficult.


1 All asm in this Chapter uses Intel notation – opposed to AT&T notation; Intel one is also the one provided by MSVC.
2 saving for using different registers
3 in MSVC17, for some reason __LINE__ is not a constexpr; however, a workaround is possible (see [[TODO]] for implementation)
4 defined as “whatever we’re writing manually”

 

Take 1 – Naive XOR

Let’s say that we have:

  • a script, which extracts a list of all the instances of obf<> and OBF() out of our source files (what we need is just a list of (type,ID) tuples, which is trivial to get)
  • another script, which takes this list of (type,ID) tuples and generates randomized instances of obf<type,ID> wrappers.

One extremely simplistic example of such a generated wrapper would look as follows:

//Listing 29.obf.xor
//GENERATED RANDOMIZED CODE
//CHANGES ON EACH BUILD
//DO NOT MODIFY
template <>
class obf<size_t,126> {
//NB: I am using full template specialization here,
// but it can be partial specialization too
  using T=size_t;
  T val;

  template<class T2>
  void init(T2 v) { val = v ^ 0x28f3472a; }

public:
  obf() {}

  template<class T2>
  obf(T2 v) { init(v); }

  T value() const { return val ^ 0x28f3472a; }
  operator T() const { return value(); }

  template<class T2>
  obf& operator *=(T2 t) { init(value() * t); return *this; }
  obf& operator ++() { init(value()+1); return *this; }
  obf operator ++(int) { 
    obf ret = obf(value()); 
    init(value()+1); 
    return ret;
  }
  //do the same for –-

  template<class T2>
  bool operator <(T2 b) { return value() < b; }
  //do the same for +, -, *, /, %
  // and IF you need them – for bitwise ops too
};

Now, let’s take a look at disassembly. There, we’ll see that while in Debug mode all the XORs with 0x28f3472a are present – in Release mode they’re completely eliminated by compiler, so our inner loop is still exactly the same, with only imul and inc inside (!).

Does it mean that at this point, we failed to obfuscate our data and code? Sure. Does it mean that any such obfuscation is hopeless? Certainly not.

Take 2 – Adding Global volatile

If we go a bit further and generate our wrapper along the following lines:

//Listing 29.obf.xor.global
//GENERATED RANDOMIZED CODE
//CHANGES ON EACH BUILD
//DO NOT MODIFY
volatile size_t global126 = 1;

template <>
class obf<size_t,126> {
  using T=size_t;
  T val;

  template<class T2>
  void init(T2 v) { val = (v ^ 0x28f3472a) * global126; }
  //Order is IMPORTANT here: if we first multiply,
  //  and then XOR – compiler will still be able to
  //  eliminate paired XORs
  //Also, using DIFFERENT operations is IMPORTANT too;
  //  otherwise, compiler may reorder and eliminate things

public:
  obf() {}

  template<class T2>
  obf(T2 v) { init(v); }

  T value() const { return val ^ 0x28f3472a0; }
  operator T() const { return value(); }

  //other operators are the same as above
};

Holding our breath, we take a look at the asm, and can see that now we’re talking! Our innermost loop looks as follows:

;Listing 29.obf.xor.global.asm
l1: imul eax,ecx
inc ecx
xor ecx,28F2372Ah
imul ecx,dword ptr[global126]
xor ecx,28F2372Ah
cmp ecx,dword ptr[x]; on-stack variable
jb l1

While it is certainly not obfuscated enough, and the idea behind the code can be still seen rather clearly, we have already managed to get our obfuscation code in.

Take 3 – ADD instead of XOR

Now, let’s see what we can do with our obf<> class to obfuscate things better. First, let’s note that XOR is actually only one of the ways to obfuscate things (and is pretty poor at that, BTW – as XOR is rarely used in normal code,5 using XOR for obfuscation means every XOR will scream loudly ‘I am obfuscation!!’); also – such non-zeroing XORs may cause AV heuristics to cause false positives [RaabeBallenthin]. However, there are lots of other techniques to make things much less obvious.

Technically, what we’re looking for here, is any kind of bijection; we’ll use this bijection to convert our data from one representation into another one (and as it is a bijection, we can revert it later).6

One very simple example of such bijection is just replacing XOR some-random-constant with ADD some-random-constant; then, our init() function will look as val = (v + (size_t)0x28f3472a) * global126 and value() function as return val – (size_t)0x28f3472a0.7 After compiling such ADD-based obfuscation (mathematically – addition modulo 2^32), The innermost loop in our generated code will be already less obvious than the one with XORs:

;Listing 29.obf.add.global.asm
l1: inc ecx
imul eax,edx
imul ecx,dword ptr[global126]
lea edx,[ecx-28F2372Ah] ; a counterpart operation
;   was moved out of the loop
cmp edx,dword ptr[x]; on-stack variable
jb l1

Here, MSVC compiler did some serious optimization relying on the interrelations between our obfuscating addition modulo 2^328 + multiplication-by-global, and intra-loop increment+comparison; it certainly didn’t make things simpler-to-understand, nice job obfuscating our intentions!9

Our Listing 29.obf.add.global.asm already looks significantly better than original, but TBH, we didn’t even start any serious obfuscation stuff. Most importantly –

As we’re not writing our obf<> classes manually (instead, we have a code generator doing it for us on each build), the sky is the limit to the obfuscations we can generate.10


5 except for XOR EAX,EAX etc. which is a very obvious special case
6 actually, we can even use more generic injections, but it is a little bit more complicated, see [[TODO]] section below
7 make sure to use ONLY unsigned types here, as signed overflow is apparently Undefined Behaviour in C++ (and starting from around 2015 or so, compilers started to abuse it, so relying on it can lead to really nasty bugs <very-sad-face />). To obfuscate the signed types – just cast them into unsigned counterpart first.
8 or modulo 2^64, depending on sizeof(size_t)
9 Of course, obfuscating wasn’t a goal of the compiler, but it just so happens that serious optimizations at least for x86/x64 command set tend to result in serious obfuscation too. This includes lots of blending of different inlined functions – in the example above we cannot really say where one function ends, and another one begins; perfect! <wide-smile />
10 for time-critical code, there are also performance considerations, more on them in [[TODO]] section below.

 

Take 4 – Throw In Multiply-by-Odd

Our next very-simple-but-efficient obfuscation is related to multiplication-by-an-odd-constant-modulo-2^3211 (and reverse operation is a multiplication-by-another-odd-constant-modulo-2^32). Explanation why it is a bijection, is beyond the scope of this book (very very sketchy – it relies on a fact that any odd-constant and 2^32 are co-primes).

How to calculate inverse-constant (given original constant), is not-so-trivial, and we’ll discuss it in [[TODO]] section below, but for the time being, we’ll use a pre-calculated one. Let’s observe that 13*17*97*315643*2829487 mod 2^32 = 1.

This means that we can write our init() function using any of the 5-numbers-listed-above, and our value() function – using all the remaining 5 numbers, for example:

//Listing 29.obf.mul
//GENERATED RANDOMIZED CODE
//CHANGES ON EACH BUILD
//DO NOT MODIFY
template <>
class obf<size_t,126> {
  using T=size_t;
  T val;

  static_assert(sizeof(size_t)==4,””);

  template<class T2>
  void init(T2 v) { val = v *(size_t)17*(size_t)97*(size_t)315643; }

public:
  obf() {}

  template<class T2>
  obf(T2 v) { init(v); }

  T value() const { return val * (size_t)13*(size_t)2829487; }
  operator T() const { return value(); }
  //other operators are the same as above
};

 

When compiled it leads to the following asm code in the innermost loop of our factorial() function:

;Listing 29.obf.mul.asm
l1: mov esi,ecx
inc edi
imul esi,eax
add ecx,1F0620CBh
imul eax,esi,23144E3h
cmp edi,edx
jb l1

Well, as we can see – even without globals it doesn’t look too obvious; sure – there is a loop, and there is a multiplication within the loop, but the other stuff (such as the second multiplication) and strange-looking constants, is not really obvious (and while it is merely obfuscation, it is not “dead code” either, so it cannot be easily eliminated by dead-code analysis).


11 of course, 2^16, 2^64 etc. will also do

 

Take 5 – (kinda)-Feistel Round

The last primitive which we’ll discuss today (with many more to follow in [[TODO]] section below), is a (kinda)-Feistel-round, known from cryptography. Classical Feistel round looks as follows: we take our variable, split it into two equal parts bit-wise, then we feed the “left” part to some (not-necessarily-bijection(!)) “round function” f(), and then XOR it with the “right” part; the result consists of the “right” part (unmodified), and “left” part (XOR-ed with f(right_part)). We’ll do almost the same, just replacing too-obvious-in-the-code-XOR with addition modulo 2^32 (and using x^2 as our “round function” f()):

//Listing 29.obf.Feistel
//GENERATED RANDOMIZED CODE
//CHANGES ON EACH BUILD
//DO NOT MODIFY
template <>
class obf<size_t,126> {
  using T=size_t;
  static_assert(sizeof(size_t)==4,””);
  uint16_t right; uint16_t left;
  uint16_t f(uint16_t x) const {
    return x*x;
  }

  template<class T2>
  void init(T2 v) { right = (uint16_t)v;
  left = (uint16_t)(v >> 16) + f(right); }

public:
  obf() {}
  template<class T2>
  obf(T2 v) { init(v); }

  T value() const { return (uint32_t)((left-f(right)) << 16) + (uint32_t)right; }
  operator T() const { return value(); }
  //other operators are the same as above
};
static_assert(sizeof(obf<size_t,126>)==4,””);

With this code, the inner loop of our factorial() function happens to look as follows:

l1: mov edx,esi
imul edx,esi
movzx esi,si
movzx edx,dx
sub ebx,edx
shl ebx,10h
add esi,ebx
imul eax,esi
lea edi,[esi+1]
movzx edx,di
mov esi,edi
imul esi,edi
mov dword ptr[ebp-4],edx
mov edx,edi
shr edx,10h
add edx,esi
movzx ebx,dx
movzx edx,si
mov esi, ebx
sub esi,edx
movzx edx,di
shl esi,10h
add esi,edx
cmp esi,ecx
mov esi,dword ptr[ebp-4]
jb l1

Good luck finding our original factorial() logic here (there is still a loop-with-imul-inside, but the rest is not exactly clear to put it mildly)…

BTW, it is still NOT the-result-we’re-aiming-for; rather – it is just an illustration of the building blocks for the real stuff; stay tuned for the rest! <wink />

[[To Be Continued…

This concludes beta Chapter 29(f) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(g), where we’ll proceed to the question of “how to obfuscate things in C++17 alone, without any external compiler” – and will get ORDERS OF MAGNITUDE more sophisticated than the above – while keeping our source code readable along the lines above]]


References

[RaabeBallenthin] Moritz Raabe, William Ballenthin, Automatically Extracting Obfuscated Strings from Malware using the FireEye Labs Obfuscated String Solver (FLOSS)

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-201-declarative-datacode-obfuscation-with-build-time-polymorphism-in-c/feed/ 2
Are Top C++ Developers Migrating to Mac? https://gladguys.com/demo/wordpress/ithare_updated/are-top-c-developers-migrating-to-mac/ https://gladguys.com/demo/wordpress/ithare_updated/are-top-c-developers-migrating-to-mac/#comments Sat, 16 Dec 2017 06:39:44 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Mac Share Among CPPCON Speakers

Motivation

It is well-known that market share of Macs out there is around 6-7% (see, for example, [Clover]); if we take into account only laptops, it will be more like 10% [Tseng].

On the other hand, Mac laptops are extremely visible, and quite a few times, I was observing that:

  • in public places such as airports, share of Mac laptops is indeed below 10%
  • on the other hand, in meeting rooms – share of Macs around me is much closer to 50% (sic!).

Experiment

Of course, I am very-well aware of dangers of such observations being outright misleading (“selective memory” bias etc.), so I was sceptical of my own observation skills. It was only during a CPPCON17 that I realized that there exists a way to perform an objective measurement of this kind of stats:

As Mac laptops are indeed very visible, we can easily review all conference videos on YouTube, and calculate a share of Macs used by speakers

Indeed, the difference between Macs and non-Macs on videos is (most of the time) fairly obvious:

non-Mac Mac

Of course, it doesn’t indicate which operating system they’re using, but hardware can be identified rather reliably.

Results

Recently, we spent a few hours doing it for last 3 years of CPPCON – and got the following results:

SPEAKERS USING: Macs non-Macs Unknown Total1 % of Macs1
CPPCON2015 24 54 4 78 31%
CPPCON2016 33 62 5 95 35%
CPPCON2017 49 64 11 1132 43%

1 excluding ‘Unknown’
2 at the time of our analysis, there were fewer videos published on YouTube than now, but we don’t expect additional data to skew the stats

 

Conclusions

I know that any conclusions and speculations in this regard are extremely risky (I will be beaten hard both by Mac fans and Windows fans, not to mention Linux fans just for a good measure); still, IMO from the (semi-scientific) results above we can make the following observations:

  • share of Macs in use by top C++ developers3 during their presentations on CPPCON, is much higher (like 6x higher) than average market share of Macs
    • Once again – it is ONLY about hardware, with no indication of the OS they’re using.
    • One may speculate that it might indicate that as soon as C++ developer reaches the point when Mac price is no longer too significant, s/he’s more likely to use Mac over non-Mac box. In other words – it might indicate that C++ developers tend to prefer non-Macs for benefits/price ratio, but tend to switch to Macs when price is no longer that much of a concern.4
  • share of Macs in use by top C++ developers, has indications of growth over time
    • In other words, there are some preliminary indications that top C++ developers tend to migrate to Mac hardware over time. This goes in contrast with overall Mac share reportedly shrinking (from ~7% to ~6% over last few years),

I am done for the day. Let the flame begin! <wink />


3 I think we can say that “those who speak on CPPCON” is a rather representative sample of “those we can name ‘top C++ developers'”
4 Yes, I know that I will be pummeled particularly hard for saying this

 


References

[Clover] Juli Clover, Apple's Mac Shipments Down in Q3 2017 Amid Continuing PC Market Decline

[Tseng] Kou-Han Tseng, TrendForce Reports a 0.7% YoY Growth beyond Expectation for 2017 Notebook Shipments; Apple Exceeded ASUS to Rank No. 4

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

 

]]>
https://gladguys.com/demo/wordpress/ithare_updated/are-top-c-developers-migrating-to-mac/feed/ 3
Bot Fighting 103. Code Integrity Checks, Code Scrambling https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-103-code-integrity-checks-code-scrambling/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-103-code-integrity-checks-code-scrambling/#respond Tue, 12 Dec 2017 16:10:56 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 You're under arrest for asking the right question at the wrong time
#DDMoG, Vol. VIII

[[This is Chapter 29(e) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

The next section in this epic chapter on Bot Fighting is related to techniques which mess with our own binary code. In general, as we’re messing with our code, this class of techniques tends to be less in the face of the attacker than outright asking OS to do things, and as such – tends to survive attacks a bit better. Still, the devil (as always) is in details, and unless we integrate protection with the rest of our code really well – we’ll end up in a protection which can be disabled in one single point (and as a result, won’t last long).

Code Integrity Checks

One simplistic way of seeing that our executable is still intact, would be to:

  • Have a function checkIntegrity() which calculates checksum of our code in-memory (more specifically – of our code segment in RAM) – and compares it to some-value-stored-in-data-segment (let’s name it codeChecksum); for the time being – let’s fill codeChecksum with 0xFFs.1
  • Compile our PE executable with checkIntegrity() and codeChecksum in it.
  • Calculate checksum value (let’s name it precompiled_checksum_value) of our code segment in compiled PE executable (for Windows, code segment usually looks as .text section in PE .exe); of course, the algorithm of checksum calculation should be exactly the same as the algorithm in checkIntegrity().
  • Patch our .exe so that codeChecksum corresponds to the precompiled_checksum_value
    • To do it, we can use linker map.
    • We will be patching .data or .rdata section, so there won’t be any issues with circular dependencies.
  • Bingo! Each time we run checkIntegrity(), we can be sure that our code is not modified.

This approach might even work; however – it has a very serious flaw, which decreases its anti-bot-writer efficiency very significantly:

The protection above has one single point where it can be disabled.

Fortunately, this flaw is not inherent for integrity checks, and it can (and SHOULD) be addressed. The key improvement in this regard is to replace one single .exe-wide integrity check with multiple per-function integrity checks. One way to implement per-function integrity checks is described in [Kulchytskyy], and goes along the following lines:

  • We have a function CalcFuncCrc(),2 which takes two pointers (beginning and end of the function).
  • We have DebuggeeFunction(), which integrity we want to check.
  • We have a fake and empty DebuggeeFunctionEnd(), which resides right after DebuggeeFunction() in code.
    • NB: here, we’re essentially relying on compiler and linker to put the functions in exactly the same order to .text section, and on each function being one monolithic block of binary code.
      • As [Kulchytskyy] notes, for our assumptions to stand, we have to use /INCREMENTAL:NO linker option (otherwise layout of the generated code will look differently)
      • Actually, modern compilers can easily generate functions which are not represented by a monolithic block; still – it won’t cause our code to fail (checks for such a checksum will still work 100% of the time, it is just that this checksum will cover a-bit-different-piece-of-code from what-we-might-intuitively-expect, which is usually good enough).
    • Whenever we want to check integrity of our DebuggeeFunction() (which integrity can be violated as a part of hack or by debugger inserting a breakpoint) – we call CalcFuncCrc(), feeding it pointers to DebuggeeFunction() and DebuggeeFunctionEnd() as parameters.
      • Then, we compare the result of the CalcFuncCrc() with a pre-calculated global constant.
    • To prevent inlining of the DebuggeeFunction()[Kulchytskyy] uses #pragma auto_inline(off)
      • From what I’ve seen, __declspec(noinline) is a more reliable way of achieving it; however – make sure to read on randomized-checks technique below, which allows to avoid disabling inlining most of the time.
    • Bingo! We’ve got an ability to have lots of different per-function checksums in our code.

IF we put lots of such checks (of different functions) into our Client – it will make life of the attacker significantly more difficult. Still, there is a significant room for improvement over the technique shown in [Kulchytskyy]. In particular:

  • In [Kulchytskyy], CalcFuncCrc() function (unless it is very small or force-inlined) can be compiled as non-inlined regular function; this, in turn, will allow the attacker, after identifying this function once, to identify all the places of interest (and this is a serious weakness).
    • To deal with it, we can either force-inline CalcFuncCrc(),
    • Or (even better) we can (and SHOULD) have different functions to calculate function checksums (ideally – these checksum functions should be generated per-function and per-build automatically).
  • With [Kulchytskyy], we have to specify manually which of the functions we want to protect – and to jump through the hoops to enable this protection; as a rule of thumb – this is going to lead to too-few-functions-to-be-protected.
    • To deal with it, we can (and SHOULD) make our code generation completely automated, and without any changes to source code either.
  • With [Kulchytskyy], we have to force no-inlining of the functions of interest, which has significant performance ramifications. It is possible to avoid it.
  • With [Kulchytskyy], the result of CalcFuncCrc() is simply checked, and a certain action is taken; as such – this action can be easily disabled (and also such pieces of code are at risk of being easily identified). We can (and SHOULD) use checks which are orders of magnitude less obvious.

To mitigate issues discussed above, the following approach will be a further significant improvement over [Kulchytskyy]:

  • We say that by default, we DON’T really care about protecting specific functions. Instead, we’re merely considering the whole code segment as a piece of data-which-should-be-constant, and are performing random checks over it.
    • As such, we’re not even required to calculate checksums over contiguous chunks of bytes; our integrity checks can be anything from a single-DWORD-check, to a check which runs a pseudo-random generator (PRNG), uses PRNG-generated numbers as addresses within code segment, and calculates some kind of obscure (and automatically generated on each build) checksum over data-which-resides-at-those-PRNG-based-addresses.
    • That being said, we SHOULD be still able to specifically check those functions of particular interest; these functions can be specified externally to the code – and (unless they’re inlined) can be found in “linker map” – and then used to generate yet another check function.
  • In addition, we DON’T want to make our reaction to the checks too obvious – that’s why we’ll use the results of our checksum calculations as an input to our obfuscation routines (more on them below). In a very simplistic example – if we’re passing a number from one part of the program to another one, we can XOR it with a constant (and XOR again with the same constant on receipt). To integrate it with our integrity checks – we can XOR the number with the result of our checksum calculation on one side, and XOR with the pre-calculated constant value on the other side. If (as it should be) our code segment is intact – everything will work like a charm; otherwise – the program will break in an extremely non-obvious manner (and without any obvious hint on where-the-fault-has-originated).

After we do all these things, our development process will look as follows:

  • We DON’T make any integrity-check-related changes to our regular manually-written source code <phew />
    • We still DO use declarative obfuscation (as discussed in [[TODO]] section below), replacing types with their obf<type> counterparts wherever we feel like it.
  • As a part of our build process, we:
    • Decide how many of the integrity checks (and of which complexity) we want to generate
    • We generate source-level code for random checking routines (those ranging from a single read from the code segment to an obscure-checksum-over-addresses-generated-by-a-PRNG) – and integrate them into our automatically-generated obfuscation code (see [[TODO]] section below for detailed discussion of generated obfuscation).
      • Usually, obfuscation can be separated into two different routines – obfuscation and deobfuscation. We’ll use calculated-checksum-from-code-segment for obfuscation, and constant-read-from-data-segment for deobfuscation (or vice versa). For the time being, as we don’t know the value of the checksums yet – all the checksum-constants-in-data-segment will be filled with 0xFF bytes.3
    • To calculate checksums at runtime, bytes from code segment can be read in several different ways:
      • If we remove relocation tables for our .exe (which as of now, I tend to lean towards) – then we can simply read data from fixed pointer values.
      • If relocation is possible – at least in theory, we should find out where the code segment is currently loaded (can be calculated as simple loaded_offset=pointer_to_function-position_of_the_same_function_in_text_segment), and use loaded_offset as a base for all future accesses.
      • As a yet another way of reading our code segment – system calls such as ReadProcessMemory() and/or Toolhelp32ReadProcessMemory() can be used. BTW, these functions can be used to read either our own process, or a different one
        • One particularly interesting case is cross-reading (and cross-verification) by parent process and child process (especially if parent debugs child as a part of self-debugging).
    • We store algorithms of these randomly-generated checksum-calculating routines for future use.
    • We compile and link our executable (including those generated random checking routines and generated obfuscation).
    • Using those algorithms-stored-above, we calculate checksums based on real contents of the code segment (=”.text section in .exe”).
    • Using linker map, we identify addresses of the checksum-constants-in-data-segment – and patch them with the values calculated in a previous step (of course, we’ll have to fix PE checksums afterwards too).
    • Bingo! We’ve got an executable, which automagically performs TONS of integrity checks, which checks are spread all over the executable, and are extremely non-obvious too.

1 Or any other constant, except for zeros (zero-initialized stuff will go to the .bss section, where it won’t be easy to patch it).
2 NB: implementation of CalcFuncCrc() in [Kulchytskyy] has nothing to do with CRC in usual sense of the term (neither it is necessary in any way).
3 as above – it can be any constant except for zeros

 

Code Scrambling

When speaking about 3rd-party protection from reverse engineering, probably the most-popular-selling-point is so-called “code encryption”. Actually, at least 99% of the time4 the key of such “encryption” is contained within the same executable, i.e. easily available  to the attacker. And encryption-with-the-key-available-to-attacker usually doesn’t qualify as “encryption”,5 but is merely a rather sophisticated scrambling (with or without compression/packing).


4 Remaining 1% is reserved to give a chance to those-protections-I-know-nothing about.
5 at the very least, it doesn’t satisfy Kerckhoff’s Principle.

 

All-at-Once Code Descrambling

Whatever the name is – this Code Scrambling is breakable by design; however, as all the other protection techniques described to date aren’t any better – among other obfuscation techniques there is certainly a potential space for Code Scrambling too.

Simple commercial protections tend to take the whole executable, “encrypt” (actually – scramble) all the code in it, and prepend descrambler to the scrambled code. Then, on the start of the scrambled executable, it is descrambler which gets launched; descrambler code will descramble the code to RAM, and will pass execution to the descrambled code, that’s it. This naïve approach doesn’t fare well against attackers for one Big Fat Reason™:

There is a single point where the protection can be disabled.

Specifically: after the code is descrambled – it resides in memory as a whole, and memory can be dumped relatively easily. While “Anti-Dumping” techniques do exist (see, for example, [Ferrie], [Assar], and [Assar2]) – most of them are inherently targeting flaws in naïve dumpers, so at best they provide only temporary protection.

Another drawback of any scrambling techniques is that they usually contain something-which-looks-as-an-encryption-loop, and encryption-loops are known to be detected by antivirus products heuristics, and at least in theory may get your Client executable flagged; while I didn’t see such occurrences when the only-encryption-loop (without other suspicious stuff, more on it in [[TODO]] section below) got signed Client executable flagged by an antivirus – you’d better start testing your game Client under different AV products (to see whether your protection makes your Client flagged as a potential malware) ASAP, and also consider lowering entropy of your scrambled code (more on it in [[TODO]] section below).

On-Demand Code Descrambling

One notable exception (i.e. an anti-dumping technique which works) actually tries to remove that single-point-where-protection-can-be-disabled; the idea (described in [Tully]) is to have on-demand descrambling: originally, most of the code pages are effectively empty (or scrambled), and marked as “Guard Pages” (actually, any kind of page-level protection-causing-CPU-exception will do). When CPU attempts to access one of such pages, an exception is generated, which is then caught by descrambler and the corresponding code page is descrambled; after descrambling – execution of the original program is resumed.

To make things a bit further worse to the attacker – we SHOULD use different descrambling routines for different on-demand regions (and descrambling routines should be force-inlined too to prevent them being easily identified).

Attacking properly-implemented on-demand descrambling, while certainly possible, is much more complicated than simplistic dumping. Still, we have to observe that it still has a single point where the protection can be disabled. Specifically, as we have to rely on handling CPU-level exceptions, we have to register our exception handler, and such registered exception handlers (at least those which I seen), are rather easy to find in executable <sad-face />.

BTW, at least in theory it is possible to have section-level on-demand descrambling as an alternative to page-level one described above. If we manage to force linker to avoid merging of .text sections coming from different object files – then each of the sections will become easily descrambleable-on-demand; this would allow us to descramble our stuff logically (=”before we want to access it”), which in turn would eliminate the exposure of the exception handler discussed above.

Technically, such section-level on-demand descrambling seems to be doable using __declspec(code_seg(“segname”)) to separate code into different PE sections (NB: note that for serious C++ code compiled by MSVC, #pragma code_seg will still leave lots of stuff in .text segment, so we have to use much more cumbersome __declspec(code_seg(“segname”)) <sad-face />). And as soon as the code is separated into different segments – they can be descrambled on-demand rather easily; the only remaining caveat is how to identify when we need a certain part code to run – and this is game-dependent and usually not-really-obvious.

A word of caution about the section-level descrambling: while it improves over the page-based one in terms of not-having-obvious-exception-handler – it loses on separating-code-into-logical-blocks (and separating things in an obvious manner is never a good thing). At the moment, I feel that with proper separation into sections, advantages of section-level on-demand descrambling would outweigh issues-it-creates, but honestly, it is just a rather wild guess on my part. And as section-level stuff is significantly more complicated for developers (we’d have to identify dependencies in the code manually – or to write a sophisticated tool doing it for us), most likely I’d still prefer to start with page-based on-demand descrambling (moving to section-level one if the need arises).

From On-Demand Descrambling to On-Demand Keys

As noted above, on-demand scrambling is not too bad (especially if combined with different descrambling routines for different pieces of code), however, for MOGs it can be improved further <smile />. For MOGs, we can have only decryption routines on the Client-Side, and obtain keys for decryption from the Server-Side when the need arises. The idea goes along the following lines:

  • We DO have on-demand descrambling (in any of the ways described above), but unlike usual descrambling (with a key hardcoded into the Client), we DON’T store the key in the Client.
  • Instead, whenever we need certain code to be descrambled, Client requests appropriate decryption key from the Server-Side, and reply arrives as “to get whatever-you-requested, use portion of the encrypted data starting from offset O, with length L, and decrypt it with key K”).
    • To make things more interesting, Client-Side SHOULD include not only what it wants to descramble, but also why it wants to descramble it (for example, for guard-page-based descrambling Client could send to the Server-Side address-where-guard-exception-has-occurred – or even all the fields of both EXCEPTION_RECORD and CONTEXT structures of the exception).

If you can manage the complexity of handling this without key-requesting latencies killing your game,6 this approach has very significant advantages even over simpler descrambling-on-demand, namely:

  • While the key is no longer hardcoded within the Client-Side executable itself, after the key is released to the Client-Side, all we have is still “mere scrambling”. On the other hand, until the key is released, it is “real encryption”.
    • As a result, in this schema (and unlike previously-discussed ones) we SHOULD use real encryption algorithm (and not obfuscation/scrambling); the
    • OTOH, in addition to encryption – we still MAY (and IMO SHOULD) use some obfuscation/scrambling, just to make things even more difficult to the attacker.
  • More importantly, with this schema Server-Side “knows” which portions of code Client-Side attempts to descramble, and – armed with its own information about which-functions-can-be-legitimately-called-in-current-Client-Side-state (and which guard-page exceptions can be legitimately raised) – it has a potential to detect blatant hacking attempts pretty easily. Examples of potential detections include such scenarios as “hey, non-hacked Client cannot request descrambling of code page 0xABCD before the MOBA match is running”, or even “hey, there is a middle of instruction at address 0xAA00, which means that such an exception can’t legitimately happen”.
    • Moreover, if we feel particularly devious, in response to such blatant hacking attempts, our Server-Side can return different (Offset,Length,Key) tuple, which descrambles into the code which is somewhat-similar-but-not-used-in-real-game, so that attacker will spend time hacking something-having-no-resemblance-to-the-really-important-stuff.
  • Even more importantly for quite a few games out there, we can avoid revealing our code at all until player’s account is valuable; effectively – it allows to  increase attack costs for the player.
    • The most obvious example of it is not revealing payment-enabled parts of code until the player has really paid us (i.e. Server-Side won’t release the key to certain parts of the code to non-paying players). And if we identify a hack after the payment has successfully gone through – we can ban not only the hacker, but also his payment method (such as credit card, PayPal account, etc. etc.). This, along with a real-world fact that payment methods are rather expensive to change, will increase attack costs by orders of magnitude.7
    • This approach of “not revealing code until attack costs are high” is certainly not limited to payments. A less obvious example is to have your higher-profile-play which is enabled-only-for-players-with-certain-experience (from certain level, in certain tournaments, etc. etc.) have a differently obfuscated set of protocols (and differently obfuscated pretty-much-everything-else within appropriate portion of the Client-Side too). Even if Game Logic for such higher-profile play is exactly the same, different automagically generated obfuscations (discussed in [[TODO]] section below) will require attacker to break them again; moreover, this time – he’ll have to do it while being under a pressure of losing a not-so-easily-replaceable-account(!).8
    • And of course, we DON’T have to reveal all our anti-bot tools until the player reaches certain level of play (so both our risks from his attack and his costs if we catch him, are higher). Just as one example, we could delay some of the AV-like scanning techniques (discussed in [[TODO]] section below) until higher-profile status is reached by player.9

BTW, this technique of key-sent-from-the-Server-Side is logically strictly equivalent to simply shipping the-code-itself from the Server-Side. Practically – sending code tends to cause a bit less traffic (at the cost of executable being a bit larger), but by these-days-standards both these differences are usually negligible anyway.[[TODO: probably worth moving into a separate section, with a discussion of pros and cons over passing keys (code vs key ~= DLL-like vs static-like: arbitrary code vs better integration+more difficult reversing)]]


6 which can be not-so-trivial for fast-paced games. In particular, requesting key over the Internet for each code page of the rendering code isn’t a good idea. OTOH, requesting decryption key for rather slow purchase-processing code might fly, as well as requesting key for the whole rendering code.
7 Of course, if the hacker is a kind of guy who buys stolen credit card numbers on the Internet, such banning won’t help much (though see about dealing with stolen credit card numbers in [[TODO-chapter/Volume]]). However, my personal observations show that a majority of serious game hackers will not engage in such outright-illegal behavior (and reducing the number of hackers-who-are-willing-to-mess-with-our-game, certainly qualifies as a Good Thing™ for the game ecosystem).
8 in practice, separation of gameplay into layers – often used by games for very different reasons – almost-universally leads to creation of “grinding farms”, but dealing with “grinding farms” is usually not that difficult, more on it in [[TODO]] section below
9 Of course, we MUST make sure that there is a sophisticated-and-obfuscated protocol between our higher-profile-detection-module and Server-Side so that absense of higher-profile detection is trivially detected

 

Code Integrity vs Code Scrambling

As Code Integrity relies on the binary code being constant, and Code Scrambling does change the very same binary code, it is obvious that they’re at odds with each other <sad-face />.

The simplest way to use both Code Integrity+Code Scrambling is to have the code descrambled at the very start of our executable, and then all the code integrity checks will work. Unfortunately, as discussed above, this class of protection is defeated way too easily, and generally, if you have other options, I DON’T recommend it.

Using on-demand descrambling alongside Code Integrity checks is more elaborated, and is not that straightforward:

  • If we’re using page-level on-demand descrambling (with a CPU-level exception processed when an undescrambled-yet page is accessed) – Code Integrity will work normally; however, there are some caveats:
    • It will lead to descrambling of the code much earlier than otherwise necessary, which in turn will simplify code dumping <sad-face />
    • Server-Side hack detection (as discussed in [[TODO]] section above) will become more difficult.
  • If we’re using section-level on-demand descrambling (or more generally – any schema when we’re descrambling because we know we want to enable certain functionality, not because of the page access or something) – then we have to make sure that our code-integrity-checking routines are run only for that code which is already descrambled.
    • In theory, it can be achieved by building dependency trees involving different sections, and ensuring that obfuscation going to one section, checks only itself and its own prerequisites, but TBH, not only I never seen such systems. I even didn’t hear about somebody contemplating the writing one.

Code Integrity/Scrambling: Summary and Conclusions

Let’s try to summarize our findings on Code Integrity/Scrambling:

  • Code Integrity Checks are nice-to-have, though while they do improve resilience to certain classes of attacks, against a dedicated attacker they count only as a rather mild
    • With Code Integrity Checks, the best technique I know about is related to interpreting code segment as an array of constant bytes, and using those constants (and randomly generated functions using those constants) as a part of our obfuscation routines which will be discussed in [[TODO]] section below. The basic idea is that if we XOR10 certain data with a randomly-generated-function-calculated-over-the-code on one side of communication, and XOR the result with the pre-calculated-constant-which-should-match-it, on the other side – it means that any change in the portions-of-the-code-touched-by-the-function will cause the program to fall apart very quickly and in completely unexpected ways.
  • Code Scrambling techniques, depending on implementation, can be categorized into 3 groups:
    • All-at-once Code Descrambling. As it is subject to dumping (and as it can be disabled in one single place), the benefits of such protection are limited (though still, if you can get it from a 3rd-party provider, it is better than nothing).
    • On-Demand Code Descrambling – either page-based, or section-based, but with the keys available within the Client. This one is much better than all-at-once descrambling, and is always preferred to all-at-once techniques. In general, if you don’t want to write DIY protection at this level – IMNSHO, choosing 3rd-party tool with an on-demand descrambling should be your choice.
    • On-Demand Code Descrambling with On-Demand Keys coming from the Server-Side(!). This one opens a whole world of possibilities to catch the hacker (formally, adding Server-Side to the picture changes attackers from operating on “Home Turf”, which is generally winnable by the hacker, into being hacker’s “Road Game”, which is generally winnable by gamedevs – see also Vol. I’s chapter on Cheating).
      • Moreover – in most cases we can even hide higher-profile functionality (or at least obfuscation for higher-profile functionality) until the player has reached higher-profile status, and ban player’s account (or even his credit card/Paypal/…) if we find he’s hacker – which increases costs of the attack very significantly.
      • And as a side note – protection-wise Descrambling with On-Demand Keys is strictly equivalent to sending the-code-itself from the Server-Side.

Taking into account an inherent conflict between Code Scrambling and Code Integrity Checks, we have to make some not-so-obvious choices. As of the end of 2017, if I am charged with protecting a Really Serious Game™ and having a carte-blanche in this regard,11 I would probably arrange my binary-code protections as follows:

  • I’d implement page-based On-Demand Code Descrambling with keys coming on demand from the Server-Side. As it is page-based, it will be the one with guard pages and descrambling on exception; and as for keys coming from the Server-Side – I’d have a request for a key having all the information about CPU registers etc. etc. at the point of the exception.
    • To enable it for a not-so-slow game – we’d have to descramble whole bunches of related pages (identifying them won’t be trivial, but for quite a few games rather short playtesting12 will reveal these pages).
    • Server-Side would look for unusual descrambling request patterns and would feed them to our Anti-Bot Team to see if the pattern is legit – or if it clearly indicates a hack-in-progress.
    • In addition to any real encryption – for good measure, I’d use generated obfuscation (see [[TODO]] section below)
  • As for the Code Integrity Checks – I’d limit it to original-unscrambled code (or code-descrambled-on-the-program-launch)
    • I’d integrate Code Integrity Checks with data-obfuscation-discussed-in-[[TODO]]-section-below, very tightly

This would be a very good foundation (in addition to other measures discussed over the course of this Chapter); in particular, it would enable the following further steps:

  • Automated Server-Side Detection of Hack Attempts
  • Identifying pages (or sections) which should never be enabled until the player has paid – and banning payment methods on proven hack attempts.
  • Section-based Code Descrambling.
    • Building dependency hierarchies of sections, which would enable Code Integrity Checks over the whole code.

Disclaimer: while I DO believe that the approach discussed above is very viable – I didn’t have a chance to try it on a large scale, so make sure to take it even with a bigger pinch of salt than usual; ultimately – it is not the advice which matters, but the reasons behind the advice (and only you can decide whether it applies to your particular game, expertise, development team, etc.).


10 Or use whatever-other-injection-or-bijection – more on them in [[TODO]] section below.
11 well, we’re all entitled to some daydreaming once in a while.
12 or even better – replay-based testing as discussed in Vol. II’s chapter on (Re)Actors.

 

[[To Be Continued…

This concludes beta Chapter 29(e) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(f), where I finally hope to get to the serious stuff – Data Obfuscation with Build-Time Polymorphism]]

 


References

[Kulchytskyy] Oleg Kulchytskyy, Anti Debugging Protection Techniques With Examples

[Ferrie] Peter Ferrie, ANTI-UNPACKER TRICKS

[Assar] Walied Assar, Anti-Dumping

[Assar2] Walied Assar, Anti-Dumping - Part 2

[Tully] Joshua Tully, An Anti-Reverse Engineering Guide

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-103-code-integrity-checks-code-scrambling/feed/ 0
Bot Fighting 102: System-Specific Kinda-Protection. Anti-Debugger, Anti-DLL-Injection, VM Detection. https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-102-system-specific-kinda-protection-anti-debugger-anti-dll-injection-vm-detection/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-102-system-specific-kinda-protection-anti-debugger-anti-dll-injection-vm-detection/#comments Tue, 05 Dec 2017 13:56:53 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Wizard of OS
#DDMoG, Vol. VIII

[[This is Chapter 29(d) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

After we discussed the very basics in “Bot Fighting 101”, we can go a bit further and discuss an activity which is traditionally very popular among anti-reverse-engineers: using certain system-specific tricks to deter attackers.

TBH, I think that while such system-level protections are somewhat useful, their importance is grossly overstated; in particular, if you think “hey, we’ll use this neat trick and we’re done protecting” – you’d better change your mind, or your game (if successful enough) will become a cheater-fest.

The problem with abusing system-level stuff as a protection is that whatever-system-level-trick-we’re-using, has a corresponding system-level-trick-which-counters-our-trick. So, at the very best, even if we find some-completely-new-trick-nobody-knows-about-yet (which is BTW extremely non-trivial) – even in this case all we can hope is to deter attackers once – and at the very most, for a few months (usually – more like a few weeks). Moreover, all specific trickery becomes utterly useless at the very moment when it is published, so writing anything new in this section wouldn’t make any sense (and for pretty-much-everything-which-isn’t-new, counter-tricks are already long-found and long-deployed by hackers).

Nevertheless, there are several reasons to use system-level protections (even well-known ones which are discussed in this section):

  • To deter novice hackers.
  • To show a notice “hey, we know you’re trying to hack us” (without any banning1). While we do know that this notice will be disabled easily, it will make sure that we won’t ban those who accidentally launched our program under debugger (or gave up really quickly); also, such a notice might strengthen position in case of legal disputes with players.2 As we’ll refer to this notice a few times in this section, let’s give it a name – GotchaNotice.
  • To integrate them with other protections (while for the time being, such integration is not in scope, we’ll discuss it [[TODO]] section below). In general, tightly-integrated different layers of protection DO have a potential to make a life of attacker signigicantly worse than otherwise (though we should be careful not to give up one protection layer when adding another one).

In addition, let’s note that:

  • For the purposes of this book, and for system-level protection, we’ll concentrate on Windows. Similar stuff exists on other platforms too, but sorry, you’ll need to find references to anti-reverse-engineering-techniques for the other platforms yourself.
  • we will not consider time-based protections as system-level ones; in the context of MOGs with servers having an independent timer, time-based protections can be made very viable and very-difficult-to-bypass (and they’re not that much system-specific either); we’ll discuss time-based stuff in [[TODO]] section below.

With all these disclaimers in place, we can start discussing some of the most popular system-level protection techniques.


1 But mentioning that it is a violation of ToC, that in case of repeated offenses user will be banned, etc. etc. – make sure to discuss details with your legal team.
2 Once again – make sure to discuss with your legal team whether they want it or not.

 

Debugger Detection/Prevention

Arguably, a technique which is most-popular-among-novice-anti-reverse-engineers, is an attempt to detect (or prevent) debugging of our program. Very often, reasoning goes along the lines of “hey, if we can prevent debugging our program – we’re done, nobody will be able to reverse-engineer it”. There is one problem with such a naïve logic, and it is that it is fundamentally impossible to prevent (or detect) debugging, at least as long we’re staying on one single box. In other words –

For each and every debugger detection/prevention trick, there exists a counter-trick, disabling our protection.3

BTW, most (though not all) of the common anti-debugging trickery can be found in [Ferrie]; while it is a bit outdated, for our purposes (as discussed above, we’re not aiming to list that-not-yet-known-to-attackers-way-of-messing-with-Windows4) it will do.

Let’s note that [Ferrie] takes over 140 pages, which means it is a lot of material; as such – we won’t discuss all the trickery in detail in this book, rather providing a very short outline – and referring to the source where a more detailed description can be found (mostly [Ferrie], but there will be some other references too). Also, for code examples for some of the techniques, you may want to look at [al-khaser]5


3 The only exception I know at the moment, is time-based detection on the Server-Side; we’ll discuss it in [[TODO]] section below.
4 by the time you read this book, it will become known to hackers anyway
5 Actually, al-khaser can be seen as a ‘sample malware’, which illustrates my earlier point on “we’ll be using quite a few techniques which are typical to malware guys” <sad-face />

 

Are You Trying to Hack Me? Honestly??

One family of extremely-popular-and-never-working anti-debugging tricks revolves around using one of the OS-level functions which are designed to tell you whether you’re being debugged. There is one problem with using these functions: when using them, we’re merely asking OS “whether we’re being debugged”, and moreover – as OS is under control of the hacker,6 we’re essentially asking the hacker “are you hacking us (please be perfectly honest when you’re answering)?”.

This means that none of those “protections” can really work in the wild; in fact,

all the protections in this subsection are disabled in bulk by easily-available-to-everybody [ScyllaHide].

Even with this in mind – we have to note that some of these non-protections are even worse than the others. Let’s discuss some outright silly stuff – an attempt at protection, which is sooo obvious, sooooo well-known, and soooo easy to disable, that it’s practically useless (except for showing GotchaNotice discussed above). This wannabe-anti-debug-protection is to call IsDebuggerPresent(), check return code and pray really hard that nobody got past this “defence”:

if( isDebuggerPresent() )
  ExitProcess();

This “protection” is in fact soooo silly that it can be bypassed literally within 5 minutes (see [DutchTechChannel@YouTube] for a demonstration). Very briefly – it is a good example of a really bad protection because of the following deficiencies:

  • Call to IsDebuggerPresent() in the executable is very obvious.
    • To mitigate this problem,7 instead of calling IsDebuggerPresent(), we MAY read “BeingDebugged” field from “Process Environment Block” a.k.a. PEB directly (see [Ferrie] for details)
  • That check after calling IsDebuggerPresent() exists only in one place, and can be trivially disabled (just by replacing ExitProcess() with NOPs in your .exe).
    • To mitigate this problem, instead of making an explicit check, we may assume that the “correct” value of the flag (which is 0 for BeingDebugged) is a constant, and use this memory location as a supposed-constant for our obfuscation routines, as discussed in [[TODO]] section below; the very basic idea here goes along the lines of “if we XOR8 our data with the value of BeingDebugged, then if we’re not being debugged – obfuscation will work as expected, otherwise – it will produce wrong result which will crash the program sooner rather than later”. This would allow us both to have this de-facto check spread all over our executable, and to get it less obvious
  • Unfortunately, even if we read BeingDebugged flag from memory directly instead of calling IsBeingDebugged(), and spread its use all over the executable, this “protection” can be still disabled by merely writing 0 to PEB <sad-face />.
    • Even worse, there is a standard open-source tool which does exactly this [ScyllaHide] <very-sad-face />
  • As a result – while we MAY use this kind of checks either to improve our obfuscation just a tiny bit, or to issue GotchaNotice discussed above – real value of such protections is negligible <very-sad-face />.

Pretty much the same reasoning goes for other similar techniques, including, but not limited to (for discussion of these techniques, unless specified otherwise, please refer to [Ferrie]):

  • NtGlobalFlag
  • Heap Flags
  • NtQuerySystemInformation().{SystemKernelDebuggerInformation|SystemProcessInformation}
  • NtQueryInformationProcess().{ProcessDebugPort|ProcessDebugObjectHandle| ProcessDebugFlags|ProcessBasicInformation|ProcessBreakOnTermination| ProcessHandleTracing} (the last two options are briefly mentioned in [ScyllaHide.doc])
  • NtQueryObject().{ObjectAllTypesInformation|ObjectTypesInformation|ObjectTypeInformation}
  • NtSetDebugFilterState() being successful (NB: if trying to play with it – make sure to read [Ferrie], as it is probably overbroad; anyway, [ScyllaHide] defeats it).
  • EnumWindows, EnumThreadWindows, and NtUserBuildHwndList [ScyllaHide.doc]
  • FindWindow(), FindWindowEx(), NtUserFindWindowEx()
  • GetWindowThreadProcessId(), NtUserQueryWindow()

Once again – I’d rather not bother with doing these kinda-protections, except for (a) showing GotchaNotice, and (b) if direct memory read is possible instead of issuing a system call (such as for IsBeingDebugged(), NtGlobalFlag, or Heap Flags) – there is some value in mixing data-directly-read-from-memory as a supposed-constant in our obfuscation routines (which will be discussed in [[TODO]] section below).


6 well, unless it is a not-hacked-yet-console
7 that is, unless we DO want it to be very obvious just to show GotchaNotice
8 any other mixing function will do too

 

Using System Calls to Mess with Debugger

In addition to reading stuff from the OS, there are also certain ways to make some system-level calls, aiming to make a debugger’s life more difficult. Unfortunately, not only all of them can be bypassed, but most of them can be disabled by the same [ScyllaHide]. Such easily-disabled kinda-protections include:

  • All kinds of messing with PE/PE headers (one example I still remember, was about marking executable as “big-endian” in PE headers – and for a few months, it even did help). As of 2017, I don’t know of any such technique which is still not thwarted by a serious reverse-engineering-oriented debugger.
    • As a rule of thumb, we still DO want to remove relocation table from our Client-Side executable (just to avoid giving up information which we can hide); also, as discussed above – we DON’T want to use DLLs where removing relocation is much more controversial
      • One downside of removing-relocation-table-from-.exe is that messing with PE MIGHT count as “suspicious” by some AVs heuristics, but as of 2017, simple removing relocations still seems to pass.
      • NB: on the Server-Side, you SHOULD leave relocation tables intact, as they enable ASLR, which is important from security perspective.
  • NtSetInformationThread(ThreadHideFromDebugger), and related NtCreateThreadEx(THREAD_CREATE FLAGS_HIDE_FROM_DEBUGGER) (the latter mentioned in [Kulchytskyy]).
  • NtSetInformationProcess().{ProcessBreakOnTermination|ProcessHandleTracing} [ScyllaHide.doc]
  • BlockInput()
  • Floating-point exceptions which used to crash OllyDbg.

In general, those system calls mentioned above are well-known, and are easily disabled by ScyllaHide, so generally I don’t recommend using them. OTOH, the following system calls are trickier and are still reported to work to confuse at least for some of the debuggers:

  • SwitchDesktop()
  • RtlProcessFlsData()
  • Disabling/patching functions ntdll.dll!DbgBreakPoint() and/or ntdll.dll!DbgUiRemoteBreakin() [Assar] (see also BlockAPI() from [belette321@ragezone.com] for a more generic example of system-DLL-patching code)
  • Self-debugging (at least in Windows, for any process there can only be one user-mode debugger). Reportedly can be disabled via setting of EPROCESS->DebugPort to 0 [Tully], but it requires messing with kernel mode, which is not that easy. In any case, self-debugging can be disabled by debugging parent executable, and running target process pretending that your debugger is that process; OTOH, to the best of my knowledge, this is really messy.
    • BTW, to make life of hacker-trying-to-hijack-parent-executable, more difficult, in conjunction with self-debugging I suggest to work on active (i.e. with messages exchanged on regular basis) heavily-obfuscated protocol between parent (debugger) process and a child (debuggee) process to complicate debugger-pretending-to-be-parent class of attacks
    • As a side benefit, the same parent-child construct can be used to measure timings in a cross-process manner (i.e. if timings between two processes on the same box are off by several seconds – something is certainly wrong in this picture – such as ScyllaHide trying to hide time-based intra-process debugging).

Trying to Detect Debugger Side-Effects

Another bunch of techniques revolves around detecting side effects of debuggers. As these side effects are significantly less obvious than a simplistic call to IsDebuggerPresent(), some of them may stay around for a while (and even better-for-us – some of them may be pretty difficult to fix for an automated tool such as ScyllaHide).

Still, the following techniques are either automagically disabled by ScyllaHide (and therefore, aren’t good):

  • NtClose()
  • OutputDebugString()
  • SwitchToThread(), NtYieldExecution()

Some of side-effect-based techniques-which-still-reported-to-work at least for certain debuggers, include:

  • Debugger detection in TLS callback.
    • In particular – it is possible to detect debugger thread via NtQueryInformationThread() in TLS callback (see [Ferrie]).
  • SetUnhandledExceptionFilter() and causing an unhandled exception.
  • LoadLibrary() call leaving the file open (this technique essentially relies on bug in the debugger, but such buggy debuggers are reportedly still in existence).
  • Guard pages via VirtualProtectEx()/VirtualAllocEx()
  • RaiseException() and/or issuing INT 3H and/or issuing strange instruction such as LOCK CMPXCHG8B [halsten@OpenRCE]. The idea is to raise various exceptions and to see whether debugger consumes them.
  • Arguably the most interesting and subtle ones – PAGE_EXECUTE_WRITECOPY (see  [https://waleedassar.blogspot.co.at/2012/09/pageexecutewritecopy-as-anti-debug-trick.html] for details, and don’t forget those #pragma comment(linker,”/SECTION…”) and #pragma code_seg(“xyz”) from the demo source code), and its close cousin, QueryWorkingSet().ShareCount [https://waleedassar.blogspot.co.at/2014/06/sharecount-as-anti-debugging-trick.html].

Still, it is IMO a matter of time until ScyllaHide includes protection from these (or debuggers fix their issues which allow for detection in the first place).

Messing with Binary Code to Cause Debugger Issues

One more bunch of techniques tries to get our binary code unusual enough so that debuggers will behave strangely when running over it. For x86/x64, these techniques include:

  • Adding REP prefix to non-string instructions. Efficiency of this technique relies on bugs in debuggers, which are rare now.
  • Using INT 2D. Efficiency of this technique relies on bugs in debuggers.
  • Using long-form INT 3 instruction (“CD 03”) to disrupt breakpoints. Current efficiency – unknown.
  • Using undocumented opcode 0xF1 [Falliere]
  • PUSH SS/POP SS to detect single-stepping. Current efficiency – unknown.
  • Using “nanomites”. Very briefly – “nanomites” use (single-byte) INT 3 instruction to cause an exception, then all INT 3 exceptions are caught, then checked whether the exception is one from legitimate exceptions in the table-stored-in-executable, and then jumping to the destination-from-the-table. Still a rather efficient technique.
Overall, these techniques (especially nanomites) are rather powerful, and do provide significant protection. On the other hand, as any single protection measure, on its own they’re certainly not sufficient to protect. On the third hand (yes, we, hares, have three hands <wink />) – it is usually done as a post-compile processing by 3rd-party tools, so normally it won’t interfere with our source-level efforts.

CC-Scanning

One popular kinda-anti-debug-protection measure is to scan our own code for INT 3 (0xCC), which is normally used by debuggers to set breakpoints [Falliere]. IMNSHO, it is a pretty flawed technique (both causing false positives due to legitimate occurrences of 0xCC in the code, and false negatives – as debuggers can use something such as 0xFA for software breakpoints [Falliere]). I suggest to leave CC-scanning alone, replacing it with much-more-reliable checksum-based integrity checks (which will be discussed in [[TODO]] section below).

Production Bot Detection/Prevention

After we discussed debugging, we can (and SHOULD) take a look at how the real (production) bot will work while our game is running.

Anti-DLL-injection

Under Windows, one of the major ways for bots to mess with our programs is known as “DLL injection.” I won’t go into detailed discussion on DLL injection, just noting that:

  • To know what we’re dealing with, make sure to read [Darawk@Blizzhackers] (especially “Code Cave” method, which is particularly nasty).
  • TLS callbacks (discussed above) will allow to detect new threads being launched within our process, including those ones injected. In general – while this protection can be disabled, it is going to be quite an effort.
  • Disabling LdrLoadDll() (via writing RET instruction to the beginning of it in your own virtual space) has been reported to work against some of the methods of DLL Injection [belette321@ragezone.com]. OTOH, as not all of injections rely on LoadLibrary() (in particular, as we’ll see below, we can kinda-LoadLibrary ourselves), so it is not a really bulletproof solution.
    • On the plus side, this technique is going to play nicely with our previously-stated “noDLLs” rule, but still may require quite a bit of tweaking to make sure that all the necessary stuff is loaded before we invoke this really-nuclear option

Also, let’s keep in mind that it is perfectly possible to manipulate our program without DLL injection (reading its memory from outside, and sending input events as if they’re sent by player). Still, anti-DLL-injection can be seen as a tool important-enough to bother about it a bit.

Anti-VM

An ultimate way of bot running undetectable is to run a VM hypervisor on the Client PC, then run our Game Client within guest, and to run the bot directly in the host. With such bot configurations, it is extremely difficult to detect bots, however:

  • At least in a theoretical Blue-Pill-vs-Red-Pill sense, with MOGs we do have an upper hand in this battle (because we have an MOG Server as an external reference, it is possible to find out whether the observed reality is simulated or real, at least in theory).
    • This logic, however, doesn’t apply to bots which DON’T modify Client-Side reality by illicit means (i.e. VM-based bots which merely read RAM of our Client, and issue mouse clicks to our VM, in theory can be made perfectly undetectable).
  • In practice, anti-VM bot fight happens to be much more down-to-earth.
    • One popular way of detecting VMs is by looking at device drivers (all the VMs I know have rather specific drivers, which can be identified rather easily); also such things as registry keys, adapter names, files, and processes (the latter – mostly if VM-supporting tools are installed into guest) tend to give VM away.
      • A whole bunch of VM detection techniques based on this kind of stuff, can be found in [al-khaser]. Hint: when using techniques from [al-khaser], make sure to obfuscate all the constants – and to integrate results of your checks into obfuscation too; otherwise, finding and disabling your protection won’t be too difficult.
    • A CPUID-based technique for x86/x64 virtualization detection can be found as a part of virt-what [virt-what]; while the code is GCC-oriented, converting it into MSVC isn’t too difficult.
    • We can also try to communicate with VM host, using tools-provided-by-VM for this purpose (see [Bachaalany] for Virtual PC and VMWare examples)
    • What to do when you DO find that Client runs under VM, is a different story. Some games simply prohibit running Client under VM (issuing a relevant message). Alternatively, you may allow running your Client under VM – but raise a “red flag” (which in turn can be used to run more checks – both Client-Side and Server-Side ones; more on it in [[TODO]] section).

In general, VM is a major threat which bot writers use to circumvent our defences, so they SHOULD NOT be left without attention.

Summary on System-Specific Anti-Debugging Measures

My recommendations for system-specific anti-debugging measures would go as follows:

  • NB: I do NOT consider time-based protection and integrity checks a part of “system-specific anti-debugging measures”, so summary below doesn’t apply to them. Time-based protection will be discussed separately in [[TODO]] section below, and integrity checks – in [[TODO]] section below.
  • DON’T assume that any of these measures will prevent debugging; at most – we’re speaking about obtaining some kind of delay.
    • To make this delay matter – we have to generate something new for each new build; as one example – we can integrate not-so-obvious debug detection with our obfuscation as discussed in [[TODO]] section below.
  • DON’T spend more than 10% of your overall anti-bot-fighting time budget on system-specific protections. I know first-hand that delving into these things can be very tempting (and can easily become a story of never-ending improvements), but their efficiency in the wild tends to be pretty low, so I strongly suggest you concentrate most of your anti-bot-fighting efforts in other fields (which will be discussed at length below).
  • DO use something outright silly simple such as IsDebuggerPresent() to issue a GotchaNotice (though make sure to discuss it, and exact wording, with your legal folks first)
    • DON’T consider it as a protection – not at all.
  • DO consider removing relocation tables from your Client-Side .exe files. NB: it is not 100% black-and-white, see discussion above.
  • DON’T spend time on anything which is disabled by ScyllaHide (well, unless you happen to know how to detect Scylla <wink />).
  • IMO, most of the techniques described above are not worth the trouble; however, there are a few notable exceptions:
    • TLS-based callbacks are not that difficult to implement – and may allow to deal not only with debuggers, but also with DLL-injection used by production-level bots
    • Disabling LdrLoadDll() (to make DLL Injection much more difficult)
    • Some serious efforts to detect running under VM (though what-to-do-when-VM-is-detected, is a business-level/GDD-level decision)
    • nanomites are rather powerful (translation: when looking for your 3rd-party-protection-for-already-compiled-executable – generated randomized nanomites all over your code is certainly a Big Plus™9). Moreover, nanomites can be implemented as a post-processing of already-compiled executable, and by a 3rd party (so they won’t eat into your anti-bot-fighting time budget).

9 OTOH, if a 3rd- party protection uses nanomites only internally (to protect its own “encryption”), their value is greatly diminished; and if it uses nanomites only internally and without any randomization-on-each-build – the value of nanomites for bot fighting becomes pretty much zero (as all non-mutable 3rd party protections have already been hacked long ago).

 

[[To Be Continued…

This concludes beta Chapter 29(d) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(e), where we’ll try to discuss another popular-and-not-really-working technique, known as “code encryption” (though “code scrambling” is IMO a better name for it)]]

 


References

[Ferrie] Peter Ferrie, The “Ultimate” Anti-Debugging Reference

[al-khaser] LordNoteworthy@github, al-khaser

[ScyllaHide] Marcus Gothe, ScyllaHide

[DutchTechChannel@YouTube] How to use an AntiDebugger Protection in C++ (Prove in OllyDbg) HD

[ScyllaHide.doc] ScyllaHide v1.3 - Documentation

[Tully] Joshua Tully, An Anti-Reverse Engineering Guide

[halsten@OpenRCE] halsten@OpenRCE, Using the CMPXCHG8B with the LOCK Prefix

[Falliere] Nicolas Falliere, Windows Anti-Debug Reference

[belette321@ragezone.com] Usaful anti dll injection function

[virt-what] virt-what-cpuid-helper.c

[Bachaalany] Elias Bachaalany, Detect if your program is running inside a Virtual Machine

[Darawk@Blizzhackers] Darawk@Blizzhackers, https://web.archive.org/web/20171007061831/http://www.blizzhackers.cc/viewtopic.php?p=2483118

[Assar] Walied Assar, Debuggers Anti-Attaching Techniques - Part 1

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-102-system-specific-kinda-protection-anti-debugger-anti-dll-injection-vm-detection/feed/ 2
Bot Fighting 101: Don’t Feed the Hacker https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-101-dont-feed-the-hacker/ https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-101-dont-feed-the-hacker/#comments Tue, 28 Nov 2017 13:54:13 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Don't Feed the Hacker
#DDMoG, Vol. VIII

[[This is Chapter 29(c) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

In previous instalment of this loooong chapter on Bot Fighting, we discussed how attackers are likely to approach your game, and categorized all the attacks on MOGs into “Attacks on Silly Deficiencies”, “Specific Attacks”, and “Brute-Force Attacks”.

Now, we know enough to start discussion on “what MOG devs can do to fend off attackers”. First, and most obviously, we have to avoid those “silly deficiencies” which quite a few of those attacks are based on.

Our operational phrase for our Bot Fighting 101 crash course is “don’t reveal information beyond absolutely necessary”. In practice, while each of the efforts to stop providing unnecessary information isn’t rocket science, making sure that you covered all of them, will require quite a bit of diligence.

The following is a (non-exhaustive) list of things to be done in this regard:[[TODO: convert from text to prose]]

  • DO use Authoritative Servers
    • It was discussed at length in Vol. I’s chapter on Authoritative Servers; in short – if you’re not doing Authoritative Servers, your game is very likely to become a cheater-fest, with the only way to handle it being moved towards Authoritative Servers. Quite a few companies already learned it in a painful way, and are in the middle of a cumbersome-and-error-prone migration of their game logic from Clients to Servers.
  • DO use Interest Management (see Vol. I’s chapter on Communications for details). As discussed in Vol. I, without Interest Management you will be distributing all the information to all the Clients, which in turn enables the whole families of attacks, including wallhacks (=”seeing through walls”) and maphacks (=”lifting fog of war”).
  • [[TODO: Linux]]
  • When you’re using system calls, consider all the information fed to the system calls, as “information already in the hands of the attacker”.1 To minimize negative impact of such leaks on the game, make sure that you DON’T reveal more than absolutely necessary, in system calls and their parameters.
    • Sure, we DO need to use system calls. In particular, there is no way to avoid the following:
      • Network calls (most of the time – socket calls)
      • Graphics calls
      • User input calls
    • OTOH, the following system calls are not absolutely necessary, and using them will significantly simplify life of the attacker:
      • System-level encryption calls.
        • Whatever encryption we’re doing, can be lifted trivially (revealing our bare protocol and enabling all kinds of bots) – as soon as we’re using system-level calls to encrypt.
        • Mind you, encryption is a Really Good Thing™ to deal with hackers, it is just that it should be done by our own monolithic executable, and not by system-level DLLs.
      • System calls which do both encryption and sockets
        • The reason is the same as for pure encryption.
      • Any system-level call which deals with character-based output – at least, for any non-constant data which is related to current game state.
        • This includes such stuff as using standard OS controls for mere display purposes (Windows controls are particularly nasty in this regard), and goes all the way up to using functions such as DrawText() and ID3DXFont::DrawText().
        • The reason is that whenever you call DrawText() to show something related to current state of the game, you have already leaked significant information about current game state to the attacker.
          • The worst real-world case I know about, was a whole bunch of poker games, using standard Windows controls to show “chat window” – which, by tradition, for poker apps has all the history of the hand up-to-now. It means that attackers can get current state of the game in an easily-parseable form, by merely sending WM_GETTEXT message to the “chat window”. And as soon as they have it – they can write all kinds of “data mining” programs (which are usually prohibited by ToC as upsetting the balance of the game and scaring players), automated advisors (also prohibited by ToC for the same reasons), and just need to add sending WM_MOUSEUP messages at fixed positions within the game window to enable writing a full-scale automated bot player.
        • To address this class of vulnerabilities – the only way I know is to owner-draw everything.
          • You need a control? Write it yourself (or at least use a 3rd-party static library which does it without using system-level calls which take character input).
          • You need to draw text (for the control above or otherwise)?
            • At least – use your own bitmap fonts2 to combine your bitmapped characters into a line of text. BTW – for quite a few games, simplistic bitmaps-with-per-character-width (but without kerning pairs) often look “good enough”.
            • At most – you can use something like FreeType library to render your TTF/OTF/…-based text into bitmap – and then render the bitmap to screen via the system-level API call.
            • In any case, all the relevant calls to Windows API will look as a bunch of BitBlt() calls, which are much more difficult to reverse engineer than reading parameters of a simple call to DrawText().
          • While we’re on a subject of 3rd-party calls: DON’T reveal information by using 3rd-party DLLs. While using system DLLs is inevitable (to the extent discussed above), using non-system DLLs within your Client is a Big No-No™. Very briefly, the reasoning goes along the following lines:
            • There is no reason to use DLLs on the Client-Side (except for official 3rd-party plug-ins for your app). For detailed discussion – see, for example, [NoBugs2010].
            • with each DLL, we’re splitting our monolithic executable, and they are providing lots of information to the attacker. And as using DLLs in 2017 (beyond some very narrow cases) doesn’t make any sense – replacing all-DLLs3 with static libs clearly qualifies as a Very Good Thing™.
              • I cannot even count the number of games which are using openssl.dll – and had their protocols hacked using this attack vector as a result.
            • Note that with F.L.I.R.T., even 3rd-party static libraries are vulnerable – but dealing with F.L.I.R.T. is a subject of a separate discussion in [[TODO]] section below.
              • For the time being – just make sure that you’re compiling 3rd-party libraries from source; in addition, when doing so, you SHOULD:
                • Use your own compiler settings (different from those used in default compile)
                • Use all the library-provided #defines to switch off all the features you don’t need within the library.
                  • This tends to play very nicely with TLS libs such as OpenSSL: as for MOG, we’re controlling both sides of communication – we can (and SHOULD) disable all the algorithms except for one we’re using (for discussion on which-exactly-TLS-algo-to-use – please refer to Vol. IV’s chapter on Basic Security); and the best way to disable unnecessary stuff is via library-provided #define
    • DO encrypt your traffic. If you’re not doing it – you just make your game vulnerable to fundamentally-undetectable proxy bots. While you’re implementing encryption:
      • DO check the certificate on the Client-Side (and DON’T use Anonymous Diffie-Hellman for encryption)
      • DO store certificate-for-checking within your executable (and DON’T use system-level storage of root certificates for this purpose)
        • DO obfuscate the certificate within your executable.
      • Along the lines discussed above – DON’T use system-level libs for encryption; neither use 3rd-party DLLs for encryption (though statically-linked OpenSSL-or-whatever-TLS-lib-you-fancy – is fine).
      • DO scramble your traffic right before feeding it to encryption library; DO it using your own method. We’ll discuss more on scrambling (and more generally – on obfuscation) starting from [[TODO]] section below, but for the time being – let’s note that such scrambling will help to protect your protocol even if the attacker manages to F.L.I.R.T. with your TLS library,4 and to get your unencrypted traffic.
        • It doesn’t matter how insecure your scrambling is (=”don’t bother to use an anywhere-standard encryption library”), the only real requirements for scrambling/obfuscation are the following:
          • It MUST be reversible for obvious reasons.
          • It SHOULD be non-standard (though it MAY be constructed out of standard primitives, more on it in [[TODO]] section below).
            • If you’re using a standard library for scrambling – it can in turn be F.L.I.R.T.-ed with, which goes against the goal of scrambling discussed right above.
          • It SHOULD be convoluted (~=”simple XOR with a constant byte is not good enough”).
            • Think of it as if you’re designing a very lightweight crypto algorithm yourself; in this process, there is a 99.(9)% chance that whatever-you5-design, won’t be secure – but it can easily be a good scrambler (and a unique one too, which is important).
          • It SHOULD have some means of integrity checking (such as “message checksum calculated in your own way and/or scrambled”). If such a checksum doesn’t match on the receiving side – it is a serious indication of being hacked,6 so it has to be reported, and probably have corresponding player flagged.
          • Ideally – your scrambler SHOULD be randomly generated for each new build out of reversible primitives (for a long discussion on it, including versioning – see [[TODO]] section below).
  • In C++, DO disable RTTI. RTTI is a nice feature of C++; however – it turns out to be  nicer to hackers than to gamedevs. If you don’t disable RTTI, than for each and every object of every-class-that-has-at-least-one-virtual-function, hacker will be able to know not only the VMT pointer for this class (and VMT pointer identifies the class7),  but also class name. Actually, this is the only case when information about C++ source-code names leaks to the executable, and it happens to be of enormous value for the attacker. Without RTTI, all attacker has is merely a “it is an instance of some class” (and has to guess about what-the-class-does); with RTTI, for the same class attacker can see “hey, it is an instance of the class which author has thoughtfully named ‘PlayerHealth’ or ‘UnitPosition'”.
    • To deal with it, three approaches are possible:
      • disable RTTI (“/GR-” switch in MSVC). This comes at a cost of losing ability to make dynamic_cast<> (which are rather easily replaced with DIY kinda-dynamic-cast based on DIY virtual functions); as for all the other goodies of RTTI such as typeid() (which might be usable to make maps of object types), I yet to see any real-world uses for them (and even if they do exist – they’re very few and far between, so replacing them with DIY virtual functions won’t be a problem).
      • in your production build, run a pre-processor which replaces all the class names with their hashes (or something similar). TBH, I don’t really like this approach (it is cumbersome, doesn’t deal with 3rd-party libraries, etc.); also – it may be interpreted as an insult to the attacker (and believe me, we don’t want to make attacker a personal enemy). If you still decide to go this route – make sure to randomize names on every build.
      • Obfuscate VMT pointers as discussed in [[TODO]] section below; as RTTI is inherently based on VMT pointer – hiding it will make debugger rather unhappy.
      • Personally, I would argue to both disable RTTI, and obfuscate VMT pointers. My rationale goes as follows: (a) hassle due to disabled RTTI happens to be very limited (at significant anti-hacker benefit); and (b) obfuscating VMT pointers is certainly a Good Thing(tm), but it can’t be applied across the board, so there going to be non-obfuscated VMT pointers in our program (and that’s when disabling RTTI at least won’t make the job of attacker easier).

Phew. The list above is certainly non-exhaustive, but following all the advice above is a de-facto prerequisite to any realistic protection. Otherwise – according to the weakest-link principle – attackers won’t even bother with hacking your other defenses (so they won’t even know how clever your other defenses are <sad-wink />).


1 As it will be discussed later – there is no good way to hide system calls anywhere reliably, which means that “we should treat them as easily-interceptable”; in other words – we should treat these calls as residing beyond-our-defence-perimeter.
2 Of course, properly obfuscated as we’ll discuss in [[TODO]] section below
3 except for those which are system-level-only; still, we SHOULD statically link msvcrt.lib etc. etc. For Mac OS, make sure to read [Kulesza]
4 =”identify functions within the library using F.L.I.R.T. feature of IDA Pro”
5 or me for that matter
6 errors in outermost encryption can be just due to packets being corrupted in transit over the Internet; however, for the data protected by outermost encryption, chances of any problem at packet level propagating below, are 2-128 ~= 3e-39. Even if we consider all the packets for all the players for a million-player game running for 10 years, we get merely 1e6 players * 20 packets/player/second * 86400 seconds/day * 365 days / year * 10 years ~= 6e17 packets, so chances of crypto randomly failing even for one single packet during this time is miniscule 2e-27 (~=”failures due to bugs etc. are much more likely”).
7 in [[TODO]] section below, we’ll discuss how to deny attacker easy access to information whether two objects are of the same class, or of different classes

 

[[To Be Continued…

This concludes beta Chapter 29(c) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(d), where we’ll get to real real obfuscation <wink />]]


References

[NoBugs2010] 'No Bugs' Hare, To DLL or Not To DLL, Overload #99, 2010

[Kulesza] Todd Kulesza, Static linking on Mac OS X

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/bot-fighting-101-dont-feed-the-hacker/feed/ 8
MOGs: Hacks and Hackers https://gladguys.com/demo/wordpress/ithare_updated/mogs-hacks-and-hackers/ https://gladguys.com/demo/wordpress/ithare_updated/mogs-hacks-and-hackers/#comments Tue, 21 Nov 2017 14:34:52 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Hackers attacking! IDA Pro, Cheat Engine, Hexinator, WinAPIOverride
#DDMoG, Vol. VIII

[[This is Chapter 29(b) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

If you know the enemy and know yourself

you need not fear the results of a thousand battles.

— Sun Tzu, The Art of War, circa 5th century B.C. —

Before we even start devising how to protect from a certain threat, we have to realize what (and who) exactly we’re facing; this is especially true when dealing with hackers. Otherwise, we’re risking to fall prey to oversimplified-solutions-which-don’t-really-work such as “hey, let’s ‘encrypt’ our code, that’s all we’ll ever need to do”.

These days, our primary adversary is somebody who has already hacked a dozen of competitor’s games, and he is keen to add our game to the ever-increasing roster of his wins. For the time being, we can ignore whether he is doing it for money, or just for fun; this, while being important in the Grand Scheme of Things™ (especially if we’re speaking about “when he decides to give up” and “how it is possible to get his hack to defend against it”), doesn’t change technical means for the attacks too much.

What is clear though is that we (as MOG developers) do NOT care about hacks by government agencies and antivirus companies;1 while, strictly speaking, the latter folks are indirectly in the picture (as in the course of their work against malware they can produce methods and tools which defeat anti-reverse-engineering protections), they rarely share their methods with sufficient-to-use-in-practice-details publicly (because of the inherent danger of their published methods being analyzed and counter-acted by malware writers) <phew />.


1 if these folks want to hack us for whatever reason – they will do it regardless of our protection; moreover, as for the former, we’d better to be hackable by them so they won’t use other means to get the information they need.

 

A Bit of History

Back in early 2000s, the whole thing was in its infancy; moreover – pretty simple techniques were sufficient to throw attackers off the scent. In [Dodd], they describe how they managed to do it by adding a checksum to their game, so they can be sure that the game is not modified. Yes –

back in early 2000s even such simple solutions were enough to delay hackers for 2 months.

Since then, attacker tools improved greatly. In particular, an advent of the concept of “interactive disassembling”, represented by IDA Pro hacking suite, has helped attackers a Damn Lot™. While the attackers did improve their attacks, changes on defending side are left much more modest, and are mostly still trying to deal only with a DRM-inspired question of “how to hide the real code from the prying eyes” – and with only very limited success too.

This, in turn, has lead to a situation that as of 2017,

All the popular protection methods lag well-behind capabilities of the dedicated attacker <sad-face />

Still, the hope is not lost – it is just we shall look beyond what-is-popular (looking for weaker spots within the attacks).

 

Hackers: Tools of the Trade

These days, a serious attacker comes to break our C/C++ game2 armed with the following set of tools:

  • IDA Pro. IDA Pro can be seen as a comprehensive suite designed exactly to hack executables. It provides both static analyzer, and debugger; while different attackers MAY prefer to use an alternative debugger, for static analysis IDA Pro is the tool of the hacker’s trade.
    • Basic stuff such as flow graphs, custom loaders to deal with “encrypted” executables, support for structs/unions, etc. etc. is already there.
    • It is interactive too – after it has done its job, you can say “hey, I identified this function (structure, union, you-name-it)” – and it will use information-you-found, for all the code. This, in turn, improves readability and allows to identify things further.
    • One of the nastier-for-us features of IDA Pro is so-called F.L.I.R.T. The idea of F.L.I.R.T. is nothing magic – they just have a list of antivirus-style “signatures” of well-known functions, and as soon as they see a “signature” of a function – they name it to show that “hey, this is actually an openssl_get_publickey() function!”. This alone can help to break our nice monolithic-and-unreadable executable into manageable-for-hacker pieces <sad-face />.
    • If you’re about to do bot fighting yourself – make sure to read [Eagle]; feeling the ways hackers think is of extreme importance to write efficient defences.
  • As for hacking-oriented debuggers, the choice is wider than just IDA; among those which are seen to be used in the wild, besides IDA, there are OllyDbg, Cheat Engine, and of course, WinDbg.
    • What we can rely on, is that whatever-debugger-is-in-use, on each step/breakpoint, it will provide the attacker at least with the following information:
      • Stack frames of the functions which are currently on the stack.
      • RTTI information (if present). This includes no less than those-class-names-right-from-the-source-code(!).
      • Even if RTTI is disabled, they will identify those-classes-with-virtual-functions via their VMT pointers (lacking names, but if hacker has identified such a class once – debugger will identify it further automatically based on the VMT pointer).
      • Let’s note that while this information is highly compiler-dependent, any decent debugger will report it for any popular-enough compiler.

In addition, there are lots of OS-level tricks which attacker may use to gather information about our program and/or modify its behavior in desired-for-them ways. In addition to static analysis and debugging, such tricks include:

  • DLL injections
  • Hooks
  • And if somebody gets really adventurous when fighting your sophisticated detection – they can go all the way into root kits, and even further into Blue Pill/Red Pill-style thin hypervisors.

BTW, we can (and often should) try to detect such intrusions and take counter-measures; however, as a Big Fat Rule of Thumb™, it will stop only relatively-novice attackers (which still has its merits, but is far from being a comprehensive protection).

As a side note: for Wirth’s sake, please don’t implement your protection as a single call to IsDebuggerPresent() and think that you already protected your executable from debugging; such “protections” are disabled way too simply, and anywhere-reasonable protection is much more sophisticated than that. Overall, the only valid reason I know to have IsDebuggerPresent() in your code – is to show a dialog box notifying the hacker that he violates your ToC (which might have positive implications from legal point of view – though make sure to discuss it with your legal department). Still, it doesn’t count as protection; the point where you wrote it, is not the end of writing protection, but rather the very beginning. For real protection stuff – keep reading, as we’ll see, there are LOTS of different things to do in this regard.


2 Hacking games written in other programming languages is different, but generally is easier (usually – much easier <sigh />), see also Vol. II’s chapter on Client-Side Architecture

 

Most Popular Attack Vectors

Now, after we described the tools hackers are armed with, we can discuss the most popular attack vectors on the game Clients. There is a long list of such attacks, so to put them into some kind of perspective I’ll try to categorize them; on the other hand – please keep in mind that this categorization is neither strict nor wide-accepted, and should be taken with even a larger pinch of salt than usual.

 

Attacks on Silly Deficiencies

There are quite a few attack vectors which exist only because we (as developers) allow them to exist; still, if we’re silly enough to allow them – we’ll be hacked (and I won’t even blame hackers for doing it). Out of these, the most popular rather-silly-from-anti-bot-perspective things which game developers happen to be very prone of doing, are the following:

  • Leaving traffic unencrypted. This enables all kinds of proxy bots, which sit between our Client and our Server, and sniff/modify things as they wish. The worst thing about proxy bots is that they can be made fundamentally undetectable. When we’re dealing with bot-which-sits-on-the-Client-box – we have a fighting chance to detect it; however, for proxy bots we’re denied even this fighting chance.
    • Unfortunately, just encrypting our traffic (as we’d do it for a web browser or any other app) is not sufficient for games. The next silly thing in this regard, is using DLL for encryption (it can be standard APIs such as sspi.dll or 3rd-party DLLs such as openssl.dll). Very briefly – there is no chance to hide DLL calls, which means that we’re giving the attacker our unencrypted protocol (and therefore, an ability to write a proxy bot) on the plate. Moreover, such attacks can be mounted even in real-time easily (see, for example, [Aiko])
      • After we eliminate DLLs from the picture – we’ll still be potentially vulnerable to F.L.I.R.T., but fighting it goes beyond simple removal of the silly stuff (more on it in [[TODO]] section below).
    • Moreover, if we’re silly enough not to do certificate check in the Client, or if we’re using Anonymous Diffie-Hellman for communications3 – attacker can mount a classical MITM attack, setting up a crypto-proxy which essentially pretends to be a Server to the Client (this connection will use the attacker’s own pair of public-private keys, which will work because of the lack of certificate checks on the Client-Side). At the same time, crypto-proxy will pretend to be a Client to the Server; this will allow crypto-proxy to forward all the traffic in both directions. As a result – attacker will be able to get his hands on our unencrypted protocol, allowing him to sniff/modify it to his heart’s content.
  • Using standard APIs to draw important text. The worst thing I’ve seen in this regard, was a bunch of poker apps using standard Windows controls to show the chat window. For poker Clients, chat window includes information such as cards dealt, player actions, and so on, so using standard controls for this sensitive information automatically enables both bot writing and data mining, obtaining the current state of the game as simple as by a call to GetWindowText().
  • Having one point where our protection can be disabled. This is especially silly when using obviously-visible standard API functions such as IsDebuggerPresent(); this combination enables very simple mountable-in-5-minutes attacks such as the one shown in [DutchTechChannel].

3 yes, I’ve seen a successful-game-doing-it(!)

 

Specific Attacks

In addition to exploiting silly things, there is a bunch of the attacks which go after one very-specific aspect of the game; such attacks are rather game-specific, but there are still some common attacks, including the following ones:

  • Texture replacement, which is used to make walls transparent, make opponents better visible against the background, etc. etc. To deal with it – keeping track of texture checksums would do the trick (but we still need to make LOTS of efforts discussed below to make sure that these texture checksums are protected and checks are not disabled).
  • Self-MITM attack. Even if we didn’t make any of the silly crypto-related mistakes discussed above, attackers can still try getting their hands on our unencrypted protocol, mounting a self-MITM attack.
    • Self-MITM is a variation of a classical man-in-the-middle (MITM) attack which was briefly discussed above; as a rule of thumb, self-MITM goes along the following lines:
      • Just as for a classical MITM, attacker generates his own pair of private-public keys for TLS encryption.4
      • Attacker finds our certificate-stored-in-Client which is normally used to prevent classical MITM. Then, within the Client, he replaces our certificate (or public key) with his own one (the one generated above).
      • Bingo! From this point on, he can make a crypto-proxy and mount MITM attack (even as our Client checks the certificate, it will check against an attacker’s certificate, enabling MITM).
    • BTW, using OS-installed certificates to validate TLS connection is usually even worse than having a certificate embedded into our Client (because it is usually trivial to add a root certificate to the list of OS-recognized ones).
    • To protect from self-MITM, we (after avoiding doing all the silly things discussed above) have to hide our certificate better, and to check its checksum too (again, hiding the checksum and checks well). As for “how to hide things” – this is going to be a huuuge subject, discussed in sections starting from [[TODO]] below.

4 BTW, even if you’re not using TLS, the attack still stands

 

Mother of All Hacks – “Brute Force” Reverse-Engineering

And last but certainly not least, we should mention “the mother of all hacks” – which is what hackers will usually do when none of the simpler ways (discussed above or otherwise), work. In a sense – it is a full-scale reverse engineering (of that-stuff-which-hacker-needs-at-the-moment). This class of attacks applies to any program, but on the other hand requires lots of effort; that’s the reason why I tend to name this family of techniques “brute-force reverse engineering”.

One common attack scenario which is next-to-impossible to prevent completely (but is still possible to delay quite a bit), goes along the following lines:

  • As we’re writing an MOG, we have to call system-level socket functions (such as WinSock functions), there is no way around it. With this in mind, attacker can identify our calls to such socket functions (as we’ll see below, there is no chance to hide system function calls completely, and even partial hiding can prove to be problematic as it might trigger AV heuristics to report us as “bad guys”).
    • Alternatively, the attack can start not from a system call, but from a standard library function identified by F.L.I.R.T. (with whatever-SSL-library-you’re-using, being one of the nastier targets).
  • Attacker gets call stack at the point where WinSock (or F.L.I.R.T.-ed) function is called.
  • Then, he traces the stack all the way up to the interesting part which causes the call.
    • In the process, to identify “what is interesting”, he has to use whatever structure/VMT/RTTI data he has access to, to understand the data we’re dealing with in those-functions-currently-on-the-stack.
      • This happens to be of paramount importance for the attackers; in fact, I don’t know of any successful attack which doesn’t use any of such data-related information to hack the program via brute-force.
      • In particular, message formats are of special interest to the hackers. I lost count of how many games were hacked using “the first 2 bytes of the message represent message type” convention. BTW, having the first field in your class Message as int type is also very popular – and is very vulnerable too <sigh />.
    • Then, attacker dissects that function-of-interest, and modifies it (or gets protocol information out of it) to get the desired result.
      • Once again, data analysis (such as structures, VMT pointers, RTTI) happens to be of paramount importance in this process.

In a sense, brute-force is the “ultimate” form of attack which can be tried regardless of whatever-we-do. Moreover, given time – any code will give up to the persistent and proficient attacker. However, as we already mentioned above – for MOGs, there is a way to deny this time to the attacker (so that by the time when he’s past our defences, the time is up and he has to start anew); how to do it – will be discussed over the course of the rest of this chapter.

[[To Be Continued…

This concludes beta Chapter 29(b) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(c), where we’ll start discussing practicalities of anti-reverse engineering protection]]

 


References

[Dodd] Gavin Dodd, Keeping the Pirates at Bay, Gamasutra, 2001

[Eagle] Chris Eagle, The IDA Pro Book: The Unofficial Guide to the World's Most Popular DisassemblerAmazon

[Aiko] Kenji Aiko, New reverse engineering technique using API hooking and sysenter hooking, and capturing of cash card access

[DutchTechChannel] DutchTechChannel@YouTube, How to use an AntiDebugger Protection in C++ (Prove in OllyDbg) HD

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/mogs-hacks-and-hackers/feed/ 4
Merits of Anti-Reverse-Engineering for MOGs https://gladguys.com/demo/wordpress/ithare_updated/merits-of-anti-reverse-engineering-for-mogs/ https://gladguys.com/demo/wordpress/ithare_updated/merits-of-anti-reverse-engineering-for-mogs/#respond Tue, 14 Nov 2017 23:31:02 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 DRM bad...
#DDMoG, Vol. VIII

[[This is Chapter 29(a) from “beta” Volume VIII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

We have already reached Vol. VIII of the nine-volume “Development & Deployment of Multiplayer Online Games” epic. By this time, if your MOG is successful, you’re already popular enough to attract cheaters. We have discussed the motivation for cheating in Vol. I’s chapter on Cheating & Authoritative Servers, so I won’t repeat it here. However, one thing has to be re-iterated:

If you’re successful enough – they will find reasons to cheat.

Even if you’re 100% sure that there is absolutely nothing for them to be gained – they will still find a way to cheat and monetize the cheat. The most glaring example I’ve seen in this regard, was a play money poker site where cheaters wrote bots to get free play chips in large quantities over different accounts, then chip-dumped them to a few dedicated accounts, and then sell these play-money chips for real dollars on eBay (so that buyers of those play chips can boast about being good players <ouch! />).

Moral of the story is that cheaters are not a function of the logic of your game – but rather a function of your game success. If you have a hundred of thousands of simultaneous players – you do have cheaters (and if you don’t see them – it is just because you’re not looking for them persistently enough).

Anti-piracy = Bad

Veg, bad.
Veg, bad.
Veg, bad.
Say no to carrots, cabbage and cauliflower.
Well, come on, lad, what are you waiting for?
Turn on the BunVac.

— Wallace, the Curse of the Were-Rabbit —

Now, let’s see our (=”MOG developers”) side of dealing with cheaters. First of all, let’s note that

while we are using a lot of the same techniques as anti-piracy guys1 – our intentions and results are going to be VERY different from them.

With anti-piracy protection, two separate issues arise:

  • From the point of view of motivation to use anti-reverse-engineering, piracy-vs-anti-piracy is a conflict strictly between pirate and developer, and there are absolutely no other parties involved. This, in turn, implies that:
    • Software companies using anti-piracy protection are often perceived as “greedy”. Among other things, pretty often, a “non-scarcity” argument is raised, going along the lines of “hey, when I copy it, you don’t lose your copy, so there is no harm done” – known as a “non-scarcity argument”. Fortunately, we don’t need to debate merits (or lack of those) of the non-scarcity argument – as for MOG cheating, as we’ll see below, there is the harm done, and a pretty obvious one.
    • Anti-piracy measures do NOT bring any improvements to the experience of the software customer, but at the same time can easily hurt the customer badly. I am not even speaking of outright stupid things such as “Sony rootkit scandal” [TODO:ref], but pretty much any usual run-of-the-mill protection is either already-broken-long-ago, or is highly-risky for the customer.
      • This means that as a customer of usual (non-MOG) software, I have all the reasons to hate any such protection (I have nothing to gain, and lots to lose).
      • As a result, I tend to agree that most of the “anti-piracy protection” measures tend to hurt producers more than benefit them.2 One potential exception to this observation can be expensive CAD programs etc. (while I am not sure about the merits of anti-piracy protection even in this case, I cannot rule them out outright). What is clear though is that for mass-market programs taking any risks with protection-making-life-of-customers-worse just isn’t worth it. In other words, with a few minor reservations I tend to agree with [TODO:Kaspersky] on that “software protection is not economically efficient”.
  • From the point of the ability to protect the software, there is a huge problem too. For any Client-Side software, we’re giving the software to the hands to the attacker (which means “given time, everything can be broken”); however, for non-MOGs it is even worse, as the hacker indeed has all the time in the world to get it broken. Moreover, as soon as the breach is found, the game is essentially over for protection (at least until next version comes out, but even when it happens, nothing prevents the whole world from using a-bit-outdated version).
    • Hey, even hardware protection in PS3 was eventually hacked – and it is not a coincidence, but a perfectly inevitable result.
    • Speaking of hardware, here goes a quote: Sergei Skorobogatov (one of the topmost authorities on hacking hardware) has wrote that “There is no such a thing as absolute protection – given enough time and resources any protection can be broken” [Skorobogatov]

As a side note: for non-MOGs, there are certain deployment schemas (essentially – when some absolutely-critical part of the task is delegated to the Server-Side), which don’t suffer from these problems (but then they don’t usually need anti-reverse-engineering protection as an anti-piracy measure); there are currently certain signs that these schemas MIGHT become a trend in the future, but for our current discussion on MOGs it doesn’t matter much.


1 Plus quite a few of our own ones
2 BTW, e-version of this very book is not DRM-protected for exactly this reason: I certainly do not want to be hated by my readers.

 

MOG Protection = Good

Right above, we analyzed the reasons why anti-piracy protection techniques have such a bad name (and why they don’t really work – except maybe for some not-so-popular and one-of-a-kind software where customers are ready to jump through the hoops to get access to it). Now, let’s see why MOGs are very different in this regard.

Motivation-wise, for MOGs anti-bot protection, there are THREE parties involved in the conflict: (a) MOG developers, (b) cheaters, and (c) all non-cheating players. In other words –

with regards to cheating, our job as MOG developers is TO PROTECT NON-CHEATING PLAYERS.

Sure, in doing it we’ll be protecting our own selfish interests too – but these will come as a side effect of the noble-goal-of-protecting-non-cheaters, so not too many people will complain. Moreover, with anti-bot protection, average non-cheating player has a LOT to gain from us dealing with cheaters, which means that in this fight, most of the non-cheating player population will be on our side,3 giving us a very powerful mandate to do whatever-is-necessary-to-do in this direction.

And from the point of view of our ability to protect – we’re inherently in a much better position than traditional anti-piracy folks. The difference is that we can update our software monthly (or even weekly)4and we can change protocols too, effectively disabling a lot of the previously-released hacks (sure, attackers can hack us again using their previously-acquired knowledge, but it is still not as easy as just doing nothing and using previous version). How to do it without spending too much time – is a subject of a subsequent discussion, for now we just have to note that it is possible (and we’ll demonstrate the feasibility a bit later).

Of course, it is not that we can provide 100% protection5 – but we DO have a fighting chance (and more importantly, even if we got hacked – we still can regroup and fight back).

In other words,

We have both a Good Reason™ and a Fighting Chance to protect our MOGs.


3 Compare it to anti-piracy when all the customer population is essentially against us.
4 The worst kind of restriction on the frequency of updates I know about, comes from Apple App Store, but even there it is not worse than 2-3 months.
5 In general, there is no such thing as a perfect protection. Even when cryptographers are saying “hey, it is not crypto, so it is not real protection”, implying that crypto does provide the real protection – under anywhere-realistic constraints it doesn’t, otherwise we won’t hear about breaches on a regular basis. Heck, even formally-proven-invulnerable stuff such as quantum key distribution, is subject to Quantum Hacking attacks (attacking not the physics, but a particular implementation behind).

 

Myths: “Don’t write abusable games” and “Anti-Reverse Engineering is a Lost cause”

Unfortunately, with lots of people confusing anti-piracy protections and MOG protections, a lot of non-arguments against protecting MOGs are brought forward to argue we shouldn’t do it (almost-universally coming from people-having-no-idea-about-MOGs). While debunking all of these myths would take too much time, I will point out two of the most popular ones.

The first popular non-argument against anti-cheating MOG protections goes along the lines of “hey, it is easy – just write only games which cannot be abused, problem solved!”. Sure, it would be an ideal solution; there is only one minor problem with it – and that it is there are no such games (at least as long as we’re speaking about at least some kind of competition). Once again – I don’t know of any single competitive multiplayer game which cannot be abused. Chess or any other strategy game? Easily abused by substituting player with a cheating computer (and these days, computers can beat humans even in Go[[TODO/wikiquote Go]]). Any team competition game with some information hidden between team players (bridge, RTS without automated sharing maps between teamplayers, etc.)? Subject to collusion by design. Anything related to reflexes/precision/… (shooters, sports, MOBAs, etc.)? Even more easily abusable than anything else.

Another popular myth about anti-cheating protection is that “it cannot be done”, which is further expanded into “we shouldn’t even try”. It is interesting to note that the most ardent supporters of this non-argument come from two completely different fields:

  • Academics. From the academical point of view, if it is “security by obscurity”6 – it is fundamentally breakable, which is (mis-)interpreted as “it is not protection at all, so we should forget about it.”. However, in practice (which is often very different from academical research) – while it is indeed “security by obscurity”, it doesn’t mean that it is useless (after all, the whole multi-billion MOG industry relies on it – and manages to strive).
    • In fact, according to the practically-observable-results (and to the logic above) – sure, there is no “once-and-for-all” solution to this problem. All we can hope for is to engage cheaters into an endless arms race – and this is exactly what we should aim to do.
      • Flash-back. As of 2005, the balance of the arms race between developers and cheaters was in favor of developers (at least those using C++ and having an idea about counter-cheating). As of 2017 – it is strongly skewed in favor of cheaters (at least for most of MOGs out there); however – adding certain MOG-specific tricks into our arsenal (which tricks will be described below) – this trend can be reversed for a while. Sure, then some new cheater will come and will find a way to break our improved defenses – but it still won’t mean that the battle is hopeless (as then we will be able to fight back at whatever-attack-he-comes-with).
    • Even in theory, there is a weak point when applying a generally-valid “given time, any Security-by-Obscurity can be broken” argument to MOGs; it is that we can deny giving attackers that time which they need to break us.7 We’ll see how it can be done, along the course of this chapter.
  • Certain companies having vested interest in protection being breakable. I remember a talk on one of GDCs, where the whole point was “hey, you can do nothing to protect your game from almighty Chinese hackers, so you should pay our company – which has Connections within the Chinese Government – to protect your game”. No further comments.

6 And what we’re doing on the Client-Side, is inevitably security-by-obscurity, there is no argument about it
7 As noted above, usually it isn’t possible for non-MOGs and anti-piracy, but is possible in MOG environment.

 

What We Want to Achieve

Now, as I hopefully managed to convince you that what we’re going to do, makes sense <wink />, let’s try to state what we want to achieve with our Bot Fighting efforts:

  • Increase attack costs.
    • In this regard, every bit counts. It has been discussed in Vol. I’s chapter on Cheating and Authoritative Servers, so just a brief recap of some related observations here:
      • In lots of practically important cases, we don’t need to run faster than the bear (=”don’t need to be unbreakable”); instead, it is sufficient to run faster than the next guy (=”be better than our competition”). After all, lots of attackers are doing it for money, so when they run into a heavily guarded system – it makes perfect economical sense for them to stop wasting their time and move to the next easily-breakable target.
      • When we have multi-layer protection – attacker should spend lots of time while breaking all of them, and usually – without any cue whether whatever-he-has-already-done-makes-any-sense, making him much more likely to give up. It means that cumulative effects of multiple protections don’t just add up in a linear fashion, but that in this case the whole is greater than the sum of its parts.
      • Different layers of multi-layer protection are especially useful if they require different skillsets (which means that one single person is not likely to break all of them); if we achieved it – we already reduced chances of the attack to be mounted by orders of magnitude.
    • Economy of hack. If we can achieve that recurring costs on our side are much less than recurring costs on the attacker’s side – we won. For example, if we can ensure that on each weekly release, we can make hacker’s life more difficult at almost-zero cost to us – it qualifies as an extremely good thing protection-wise. The key here is to change things cheap and often on our side.
    • On the way of increasing attack costs, we have to be very aware of the Weakest-Link Principle. If we have an obvious heavily guarded entrance, it is quite easy to forget about that side kinda-backdoor-which-we-intended-for-testing-only and which stuck in our code forever.
      • One obvious example of the Weakest-Link Principle in action is to have a scripting language in our Client. Whatever we do to obfuscate/scramble our scripting code – itwill be revealed, and will be used against us (usually with devastating results).
        • It means that standard VMs to interpret our scripting language on the Client Side, are pretty bad for protection.
        • On the other hand, if scripting language is handled by a non-standard VM – it can be a plus protection-wise (better still, we can compile script into our own non-standard bytecode, and then use a non-standard bytecode-VM with byte codes changing on each release).
  • Complicate collaboration between different hackers / hacker teams. Modern world tends to be highly specialized, and hacking world is no different. With a decent protection, is not that common that a single attacker breaks the whole thing. Rather, it usually goes as a series of minor breakthroughs. A typical attack scenario unfolds as follows (of course, for each specific attack phases involved will be different, but the fact that there are multiple phases, still stands most of the time): first, attacker A finds that we have our TLS certificate in our .exe XOR-ed with 0xAA; he can do nothing with this info himself – but publishes it. Second, attacker B, knowing it, mounts a self-MITM attack (discussed in [[TODO]] section below), and writes a tool which gets our under-TLS protocol exposed. Third, attacker C comes in and finds out a compression algo we’re using, exposing “bare” messages. Fourth, attacker D, armed with this information, manages to find out the format of our messages. And fifth, attacker E writes a proxy bot on top of all those things.
    • Our aim here is to complicate this kind of collaboration as much as possible. One way to do it, would be to change the-way-we-obfuscate-our-certificate, and the way we obfuscate our under-TLS protocol, for each release; in this way, information found by some hackers/teams, will become obsolete too soon to be useful for other hackers/teams. How to achieve it – is a separate question, which we’ll discuss in detail below.
  • Our code MUST NOT be identified as malicious. In modern reality, we have to live with all the kinds of antivirus products (often of questionable quality) plaguing player’s computers. And the last thing we want to have on our hands, is if several of those AV products will start to identify our game Client as a malicious software.
    • Avoiding being misidentified as malicious is not that trivial as it sounds – especially because LOTS of usual protection techniques are the same for both viruses and protections.8 This is especially true for protections involving OS-level protection, and/or code encryption. More on “how to avoid to become an AV false positive” in [[TODO]] section below.

On a more tactical level, we’ll try to achieve the following:

  • No “single point of attack”. If we have a very complicated protection, but it can be disabled in one single point – such a protection will be broken sooner rather than later. This covers all those “code encryption” schemas which fall apart as soon as the key is extracted out of your Client.
    • Instead, what we want to have, is a multi-layer protection (known as “defense in depth” in security world), plus we want to spread it all over the code.
  • Avoid relying on 3rd-party run-of-the-mill protections. If there is a 3rd-party protection, and it is not a custom one for your game only – it means that it is widely available. And the more protection is used – the juicier target it becomes, and more likely it will be broken. Another way to see it, is to observe that the more popular protection is – the higher reward/effort ratio it is for the hackers.
    • To make things worse, the vast majority (if not all) of the 3rd-party protections can be disabled in one single point, going against “no ‘single point of attack’” rule above.
    • NB: I do NOT mean that “you MUST NOT use 3rd party protections”; rather, what I mean is that “relying exclusively on a 3rd-party protection is not a good idea”. As one of the protection mechanisms – 3rd-party stuff may be ok, but IMNSHO if the 3rd-party protection takes over 20% of your overall protection – you’re doing it wrong (as a rule of thumb, most of widely-known 3rd-party protections are already broken, so disabling them is only a matter of running a ready-made script).
  • Both code protection and data protection. This is one of those things where MOGs are quite different from anti-piracy solutions. For anti-piracy, the only asset they care about, is code.9 For MOGs, data protection is of paramount importance (in fact, as we’ll see a bit below, most – if not all – typical attack vectors rely on data being unprotected).
  • Change things often at zero cost. As discussed above, ideally – we’d like to have our binary code to be changed in hundreds of different places on each release with zero source code changes. Apparently, it is possible to achieve (and we’ll see how it can be done, along the course of this Chapter).
  • Keep our source code readable. Yes, I mean it – in spite of all the obfuscation we need, our source MUST remain readable. We just cannot afford to have obfuscated-C-contest in our source code.
    • It means that we should be obfuscating binary code while keeping our source code as-close-to-non-obfuscated-one as possible. As we’ll see below, with a few tricks it is not that difficult to achieve.

8 As a side note, we may also use quite a few techniques typical for antivirus products too; these include signatures (of well-known cheating programs), suspicious hook detection etc.
9 in fact, they SHOULD care about data protection too, but this is not our problem.

 

[[TODO: Obfuscating Open Source is Not an Oxymoron (build-time randomization, should work to mitigate cheating in open-source MOGs)]]

 

[[To Be Continued…

This concludes beta Chapter 29(a) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for Chapter 29(b), where we’ll start discussing more practical aspects of anti-reverse engineering protection]]

 


References

[Skorobogatov] Sergei Skorobogatov, Physical Attacks on Tamper Resistance: Progress and Lessons

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/merits-of-anti-reverse-engineering-for-mogs/feed/ 0
Real-World 802.11ac Wi-Fi Testing: 7×6 Routers-x-Adapters Matrix. Part III. Results and Conclusions https://gladguys.com/demo/wordpress/ithare_updated/real-world-802-11ac-wi-fi-testing-7x6-routers-x-adapters-matrix-part-iii-results-and-conclusions/ https://gladguys.com/demo/wordpress/ithare_updated/real-world-802-11ac-wi-fi-testing-7x6-routers-x-adapters-matrix-part-iii-results-and-conclusions/#comments Thu, 09 Nov 2017 23:16:06 +0000 http://gladguys.com/demo/wordpress/ithare_updated?rabbit_rss_ver=23 Now, we’re coming to the juiciest part of our three-part mini-series on real-world Wi-Fi testing – to RESULTS (for test setup, please refer to the first part on the routers-we-used and the second part on the adapters).

Cold Hard Data

5GHz Results

AC8260 Dn AC8260 Up ALFA Dn ALFA Up NET-DYN Dn NET-DYN Up AC3165 Dn AC3165 Up Kootek Dn Kootek Up Supremery Dn Supremery
Up
Nighthawk X8 559,11 491,98 1,05 275,80 272,30 282,55 161,40 183,30 1,41 235,95 252,90 156,70
AirPort 394,45 305,70 77,45 50,55 312,90 306,55 148,20 107,50 76,80 46,55 148,80 284,70
ASUS-5300 612,48 460,00 1,87 260,16 431,85 441,85 159,80 182,90 0,53 201,50 170,90 143,60
ASUS-3200 578,57 362,08 2,18 213,89 334,30 453,70 170,30 184,50 3,30 189,30 25,00 134,50
ASUS-58U 204,50 387,91 1,90 260,00 356,10 476,00 184,40 184,50 174,50 157,90 221,30 128,80
WE 1326 423,79 293,72 210,94 276,73 325,53 291,58 159,40 163,40 180,15 158,05 25,00 134,90
Archer C2 224,50 231,25 195,81 237,09 215,85 258,55 168,90 182,30 107,35 151,74 140,40 101,00
Theoretical 867,00 867,00 867,00 867,00 867,00 867,00 433,00 433,00 433,00 433,00 433,00 433,00

All results are in MBit/sec; ‘Dn’ means ‘from router to adapter’, and ‘Up’ means ‘adapter to router’; as it was mentioned before – all the results are for a single TCP connection.

The same results can be seen in the following picture:

7x6 routers-x-adapters matrix. Test results @5GHz

2.4GHz Results

Similar data for 2.4 GHz look as follows:

AC8260 Dn AC8260 Up ALFA Dn ALFA Up NET-DYN Dn NET-DYN Up AC3165 Dn AC3165 Up Kootek Dn Kootek Up Supremery Dn Supremery
Up
Nighthawk X8 111,71 96,76 1,60 98,21 104,5 95,75 47,2 45,2 1,86 43,05 26,1 54,6
AirPort 92,53 113,80 77,95 61,20 102,5 106,3 97,7 66,9 46,10 35,85 38,1 56,9
ASUS-5300 195,89 212,19 203,60 176,13 28,6 21,18 82,2 84,3 1,26 81,35 118,3 95,1
ASUS-3200 205,07 167,04 2,25 0,001 138,75 129,45 86,7 78,4 125,73 92,95 113,2 96,5
ASUS-58U 204,50 185,21 1,45 172,90 190,9 183,6 84,7 82,2 101,80 82,90 105,2 40,5
WE 1326 167,24 163,07 149,44 174,33 183,2 164,8 82,1 79,2 96,85 79,25 102,6 65,4
Archer C2 145,75 102,00 120,87 145,09 120,05 114,1 77,5 85,1 100,25 98,68 103,3 95,1
Theoretical 300,00 300,00 300,00 300,00 300,00 300,00 300,00 300,00 150,00 150,00 150,00 150,00

And graphical representation of the same thing can be seen on the following graph:

7x6 routers-x-adapters matrix. Test results @2.4GHz

1 not able to connect

 

Conclusions

Now, let’s try to make some generalizations and conclusions based on the data above. As usual, everything goes with a Big Fat Disclaimer(tm) that all the conclusions are based on a limited data set, and that the picture can change in the future, and that YMMV, etc. etc. Still, IMO the following can be said with more-or-less confidence:

  • There is no such thing as “the best router”; strictly speaking – you need to test each specific pair of (router,adapter) to get anywhere-reliable results.
  • Our results for some of the adapters are better than those which were observed in [Winkle2013]; whether it indicates a difference in testing setups, or an overall improvement in Wi-Fi routers/adapters over these 4 years – is unclear, but being an optimist, I certainly hope for the latter <smile />.
    • It seems that we’ve got to the point when we can occasionally get to 50% (and sometimes even to 70%) of the theoretical bandwidth, which isn’t bad.
  • As long as we’re speaking about the trivial Wi-Fi network with laptops connected directly to routers – all 2×2 802.11ac/802.11n routers and above, are about the same. In particular, there is no obvious correlation between router price and measured speeds. Sure, it is a logical consequence of us using only 1×1 and 2×2 adapters, but as these adapters tend to dominate the laptop market by far, this choice of adapters is just a reflection of real-world realities. On the other hand, as soon as we get our clavicles on 3×3 laptop adapter – we’ll add it to the mix (but won’t add too many of them to avoid skewing the picture).
  • As for the adapters – they DO matter, and A LOT. Here, we can (still very tentatively at this point) observe the following properties:
    • built-in laptop adapters tend to outperform comparable USB adapters. In other words – think twice before buying 2×2 USB adapter to replace your 2×2 built-in adapter.
    • When speaking about the chipsets and manufacturers – there is a so-far-rather-clear hierarchy:
      • The best ones are those by Intel (AC****); however, AFAIK, Intel doesn’t produce USB-enabled Wi-Fi chips, so building a USB adapter on top of Intel chips is going to be expensive (and AFAIK, they don’t exist now).
      • Mediatek is IMO a “reasonable price/performance choice”. Also, it is “the best thing we can realistically get for USB” (in particular, NET-DYN is pretty good; the only its observed weak point was with AC-5300 on 2.4GHz, and even then @25MBit/s it wasn’t “atrocious”).
      • As for the Realtek – with that limited data we have now (and given those two Realtek adapters we’ve seen – Alfa and Kootek), I’d rather stay away from Realtek-based adapters. In 2017, connection speeds of 50kByte/s (yes, that’s KILO-bytes in the local network!) are not just “slow”, they’re “atrociously slow”.
      • NB: up to now, we didn’t test any Broadcom- or Qualcomm-based adapters (yet); my wild guess is that these should be good (at least on par with Intel) – but let’s see until we add some Broadcom- and/or Qualcomm-based adapters to the matrix.

Future Work

In the future, we hope to extend our matrix, so we have a better picture. In particular, over time we hope to add 1×1 router or two to the mix, and to extend our list of adapters (in particular, adding Killer and probably some 3×3 adapters are on our TODO-list, as well as, probably one more Realtek adapter to see whether the suggestion above stands).


References

[Winkle2013] William Van Winkle, Gigabit Wireless? Five 802.11ac Routers, Benchmarked, 2013

Acknowledgement

Cartoons by Sergey GordeevIRL from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.

]]>
https://gladguys.com/demo/wordpress/ithare_updated/real-world-802-11ac-wi-fi-testing-7x6-routers-x-adapters-matrix-part-iii-results-and-conclusions/feed/ 5