March 25, 2022

ReThinking If/Else - Use the Force (part 1)

A modern OOP approach to if/else conditional




Everyone knows IF/ELSE IF/ELSE IF/..../ELSE . And also everyone knows the evil inside a pletora of if/else_if/.... or not?
really not?
So, let these images shows the evil:






Are u convinced now? So, let's start to discover new ways to handle if/else_if/.../else logic.


Assumption 1:

often we can not avoid a conditional logic, because simply we need.

We have to compare some strings, some values against number/date/whatever.. sometimes there are multiple combination of those, using AND or OR, and so on.

So, what are we talking about? Of course not of "avoid comparison", but "improve the ways we are using to do the comparison".

Assumption 2:

scenarios covered by this tutorial cover have a common property: all if/else_if/../else act on (almost) same set of outputs, or provide 1 output of same type, such as:

  
  /* SAMPLE 1 */
  
  if (logic1) { 
    return 1; 
  } else if (logic2) { 
    return 2; 
  } else if (logic3) { 
    return 3; 
  }
  

OR
  
  /* SAMPLE 2 */
  
  if (logic1) {
    x = ..;
    y = ..;
  }
  else if (logic2) {
    x = ..;
    y = ..;
  } else if (logic3) {
    x = ..;
    y = ..;
  } else if (logic 4) {
    x = ..;
    // not assign a value for y, here...
  }
  



PART 1: comparison on ENUMERABLE


What is Enumerable ? They are values for which we could have a group/collection containing those values; Enumerables are a finite number of elements, well known a-priori, that is: when we are writing the comparison code, we already know values against we are comparing our variable.

So, we could collect these values using any of Collection types:
  • Set, which admits not null values and not multiple same values: that is, no multiples "A", "A", "A", nor 'null'
  • Map, which associates a value to a key, and the keys set is just a Set Comparison on Enumerables could be used when we could apply some simply logic on strings (simple scenario), or on some set of objects for which we could have a kind of comparison.

Use Case A: simple comparison on strings/objects

We could write something as:
  
  /* SAMPLE 3: */

  String s = ...;
  if (s.equals("A") 
      || s.equals("B")) 
      || s.equals("C")) 
      || ... 
      || s.equals("Z")) { 
    ... 
  }
  
So, really we would write a code as below? No, we would not, but often I saw a code as example above, really: once time there were 37 if/if/if/..., eventually to grow with new cases.

Of course we could use if /else if/else if/else.. but the fact is: we are doing N comparison until we will find the exactly match (if any), so our code cost O(n) and it is very difficult to read, test, maintain, enrich... because for each new case we have to modify the code.

Solution A: use Set

Solution A1: on strings
  
  /* SAMPLE 4 */

  Set<String> set = Set.of("A", "B", "C", "D", ..., "Z");
  // or, better: the set could be populated from 
  // db/configuration/any_external_source_easy_to_maintain

  // and then:
  if (set.contains(myString)) { 
    ... 
  }
  
et voilĂ : cost is O(1), because lookup into Set costs 1 operation, the code is really readable and maintanable, and if u want add further case, it's enough to add another string on configuration/db_table/whatever

Use A2: on standard Java object

If our variable is a String/Date/Integer/..., we could still use Set to collect our enumerable types, such as:
  
  /* SAMPLE 5 */

  Set<Date> dates = ...;
  // and
  if (dates.contains(ourDates)) {
     ...;
  }
  
Solution A3: simple comparison on custom object
  
  /* SAMPLE 6 */

  // let we have a custom object
  class MyObject {
    String id;
    String name;
  }

  // we could still use Set to collect our enumerable Object, such as:
  Set<MyObject> myObjectSet = Sets.of(..., ..., ...);

  // and, so:
  if (myObjectSet.contains(ourObject)) {
   ...;
  }

  // BUT, careful: our MyObject must implement Comparable, 
  // or Set will not work (AT RUNTIME!);
  // so, let our object implements Comparable:

  class MyObject implements Comparable {
    String id;
    String name;
    @Override
    public int compareTo(final MyObject o) {
      return o.id.compareTo(id);
    }
  }

  // now MyObject implements Comparable#compareTo method, 
  // and automatic comparison from Set will work using contains
  

Use Case B: comparison on strings and do something

  /* SAMPLE 7 */

  String s = ...;
  if (s.equals("A")) { doingForA }
  if (s.equals("B")) { doingForB }
  if (s.equals("C")) { doingForC }
  ...
  if (s.equals("Z")) { doingForZ }
  

Similar to A cases, but here we would do something different according to each IF.

For this scenario, we could take advantage of Map+Lambda (if we are in Java; while in Javascript the literal object could supply; C# has dictionary+delegate, etc).

First, let's define a Action interface, generics aware:
  /* SAMPLE 8.a */

  interface Action<I,R> {
    R execute(I input);
  }
  
We have an 'execute' method, accepting an input of generic I, and producing a result R; we could specify some real type, such as <String,String> or also <Void,Void> to have null input and null output, or other combination.

Then, we use a Map to associate a String (the nth string we would compare against our string) to an Action to perform if we match on that string; this pattern is called Map of Function(s), or Functor from old C++ terms:
  /* SAMPLE 8.b */

  Map<String,Action<String,String>> map = new HashMap<>();

  // then populate
  map.put("A", new Action<String,String>() {
    @Override
    public String execute(String input) {
      return input.toUpperCase() + "_A";
    }
  });
  map.put("B", new Action<String,String>() {
    @Override
    public String execute(String input) {
      return input.toUpperCase()+ "_B";
    }
  });
  map.put("C", new Action<String,String>() {
    @Override
    public String execute(String input) {
      return input.toUpperCase() + "_C";
    }
  });
  // other strings to handle...
  
  /* SAMPLE 8.c */

  // but we could take advantage of compact Lambda form and write the population as:
  map.put("A", input -> input.toUpperCase() + "_A");
  map.put("B", input -> input.toUpperCase() + "_B");
  map.put("C", input -> input.toUpperCase() + "_C");
  ...
  // nice, really? yes, very nice.
  
and then the usage:
  /* SAMPLE 8.d */

  String theInput = ...

  Action a = map.get(theInput);
  String result;
  // because we could not have any match, 
  // we have to check a 'null' value Action from Map,
  // eventually causing NullPointerException
  if (a != null) {
    result = a.execute(myInputToUse);
  }
  
  /* SAMPLE 8.e */

  // but we could (should) use Optional from Java8:

  String result = Optional
  		    .ofNullable(map.get(myStringToCheck).execute(myInputToUse))
  		      .orElse(-1);

  // where '-1' is a our default value to use as "else" case;
  
Another approach could be using (enhanced) Enum from Java5:
  /* SAMPLE 8.f */

  public class MyClass;

  private final MyService myService = new MyService(); 
  // or use @Inject/@Autowired/whatever

  // declaring
  public enum ActionEnum {
    A {
        @Override
        public String execute(String input) {
          myService.doSomething(); // but this will not work - see below
          return input.toUpperCase() + "_A";
        }
    };
    // other for B, C, ...

    public abstract String execute(String input);
  }

  // usage:
  public void myMethod() {
    ActionEnum.A.execute("asd");
  }
  
Here the function code is implicit into 'execute' method which each Enum instance implements: this could be a short approach for certain scenarios, or not.. it depends; Enum has its pro/cons: it is compiled time and statically resolved so:
  • the pro: the ActionEnum.XXX <- will not compile if you try to search an enum instance never declared; instead, using Map you will not discover on right values until you will use them, at Runtime...
  • the cons: if you declare the enum within another class (a nested enum) you could NOT pass instance fields from external class into execute method implementations, such a Closure, because Enum is static resolved by JVM; the line "myService.doSomething" will not work

Considerations:
Too much code to write to populate Map in Sample 8.bXXX? perhaps yes, surely not in Sample8c .
Too difficult to read? again perhaps if you don't know Java8 Lambda (so, it's time you study it..).
Too much costly to execute? definitely not, because lookup into Map and Enum costs O(1), so our code will not execute all if/else_if searching for matching case, but it will point directly to matching string, using lookup on map/enum, and so finally execute the internal code from Action/enum method (which, in boiler plate if/else_if, it is the code within each 'then' block).


Of course, we could (should) use an intelligent IDE (Eclipse, IntellijIdea, whatever) to have completion/aiding/etc during code writing; in example: Eclipse transforms boiler plate code using anonymous classes (8.b) to Lambda version (8.c).


Finally, we could avoid to create our custom Action interface, and, for this scenario, use directly java.util.Function, which, basically, acts as our Action:
  /* SAMPLE 9 */

  Map<String,Function<String,Integer>> map = new HashMap<>();

  // then populate
  map.put("A", new Function<String,Integer>() {
    @Override
    public Integer apply(String input) {
      return 1;
    }
  });
  ....

  // java Function provides "apply" method, instead of our custom execute, 
  // but the lambda version is the same as 8b:

  // the population
  map.put("A", input -> 1);

  // and also the same usage from 9a retrieving/executing the Action 
  // (here Function, with 'apply'), 
  // using explicit check on null or better the Optional
  Integer result = Optional.ofNullable(map.get(myString).apply(theInput)).orElse(-1);
  
java.util.Function provides other useful interface to implement a Functional Paradigm: BiFunction (2 input, 1 output), Consumer(1 input, no output), BiConsumer(2 input, no output), Supplier(no input, 1 output). If we need more *Function (or *Consumer), we could implement our own, such as a TriFunction:
  @FunctionalInterface
  public interface TriFunction<T, U, V, R> {
    R apply(T t, U u, V v);
  }
  
and use it with 3 inputs - and, of course: QuadriFunction, PentaFunction, and so on...

If we want return acts on multiple variables in our 'then' blocks, using Functor is always possible; we have just to return a Holder object:
  /* SAMPLE 10 */

  // declare our results holder, using @RequiredArgsConstructor annotation from Lombok
  // which generate a constructor for final fields
  // (check other Lombok annotations, they are very powerful: https://projectlombok.org )
  @RequiredArgsConstructor
  class MyResult {
      final String from;
      final String value;
  }

  // declare the Functor
  Map<String, Function<String, MyResult>> map = new HashMap<>();

  // populate the Functor:
  map.put("A", t -> {
    MyResult mr = new MyResult("A",t);
    return mr;
  }); 
  // or better using the best Java8 Lambda compact syntax
  map.put("A", t -> new MyResult("A",t));
  map.put("B", t -> new MyResult("B",t));
  map.put("C", t -> new MyResult("C",t));
  ...


  // finally use the Functor, returning an empty object if no matches on Map keys
  // returning an empty object is also known as "NullObject pattern"
  // check it on same design pattern tutorial...
  MyResult result = Optional.ofNullable(map.get("A").apply("someValue"))
    .orElse(new MyResult());

  // instead here we want it throws a Java standard 'NoSuchElement' 
  // exception if no matches on Map keys
  MyResult result = Optional.ofNullable(map.get("a").apply("someValue"))
    .orElseThrow();

  // while here we want it throws a custom exception if no matches on Map keys
  MyResult result = Optional.ofNullable(map.get("b").apply("someValue"))
    .orElseThrow(() -> new MyResultEmptyException("no MyResult for 'b'));
  
Well, we arrived at the end of PART_1, and we saw some technique to avoid boilerplate/annoying/unmaintanaible/etc 'If/else_if' code, when we could use a kind of comparison on Enumerable values.

In second part, we will see another approach, where no enumeration is possible, because comparison logic is not so trivial.

October 30, 2014

September 21, 2014

Faci

Faci Project

A new project is starting out: Faci, a.k.a. "FAcebook Cliques Interactions".

Faci is a Java tool allowing to analyze your facebook friendlist, and discover if people really interacts with each others, while they are in same groups - or if they are only "friends".

Source code Here

Here a movie from a running demo

And below, some screenshots, achieved applying some graph theory metrics to 'friendship' and 'interactions' graphs:

my friendship network, including myselfmy friendship network, excluding myself
my friendship network, excluding myself:
giant connected components
my friendship network, excluding myself: cliques
my interactions network, including myselfmy interactions network, including myself: cliques
my interactions network, excluding myselfmy interactions network, excluding myself and zero degree nodes
my interactions network, excluding myself:
giant connected components
my interactions network, excluding myself: cliques


Facri Privacy:
the privacy policy are available at: https://github.com/k0smik0/FaCRI/blob/master/facri_privacy_policy.md

February 1, 2014

ehfsc - the (almost) ultimate solution for traffic shaping in linux

E-HSFC a.k.a. Enhanced HFSC

Some people know hfsc, an efficient traffic classifier available in Linux kernel.
For who doesn't, it's time to do ;D

http://en.wikipedia.org/wiki/Hierarchical_fair-service_curve
and
http://www.cs.cmu.edu/~hzhang/HFSC/main.html

In 2 words: it allows link-sharing, real-time, adaptive best-effort in a hierarchical way, better than HTB or other hierarchical classifiers.

HFSC is activable via "tc" tools, well-known for its hard syntax, even when used in conjunction with iptables.


So, e-hfsc is a powerful script for people who don't want wast time in annoying stuff like tc commands:
it's just a "2-minutes config-and-start".

There are some mandatory values to set, as, i.e.:
DOWNLINK=8000 
UPLINK=320 
# Device that connects you to the Internet 
DEV=eth1
#
# other values to set, see file header
#
or, for ports you want apply shaping on, something as:
INTERACTIVE_PORTS="tcp:22 tcp:23:c udp:53:c tcp:5900,5901 udp:1190 tcp:6667-6670:c"

# where the syntax is:
# proto1,proto2:multiple_ports_comma_separated|port-range:[c/s]
# where: proto1,proto2: tcp,udp, and c/s is "client/server"
# i.e. "tcp:24650-24690:c" means "tcp ports from 24650 to 24690 as client"

# "c" and "s" are optionals: if specified, they are used only, respectively, 
# "c" for destination traffic (acting as client), 
# "s" for source (acting as server)

# so, if you have a server listening on tcp 12345 port, use "tcp:12345:s", 
# while if you have a http server, use "tcp:80,443:cs", since your server 
# surely listens on 80 and 443, but your box is also probably a client 
# for external http server (hence "c").


There are other filters available, that is "IP" and "L7-PROTO":
IP is easy to understand (I hope ;D), while "L7-PROTO" refers to same name netfilter/iptables module,
"l7-proto", that alllows to match traffic type specifying its name according to ISO-OSI layer 7 rfc,
i.e. "ssh", "ftp", "http", "torrent", and so on.
Since it comes with kernel modules, if its relative .ko file and iptables .so file could be not retrieved,
e-hfsc will simply ignore rules using l7-proto.

All (mandatory and optional) values must be setted in file header.


Finally:
./e-hfsc start|stop|restart

Check last version (and fork) at https://github.com/k0smik0/e-hfsc !

The image above show last 24h statistics of my network:
For people who want directly read the code, voilĂ :
#!/bin/bash
# encoding: UTF-8
#
# E-HFSC ("enhanced-hfsc") is a script for make better gain for voip, interactive and http 
# traffic, allowing even the p2p one
#
# heavily based on Bliziński's hfsc script, from http://automatthias.wordpress.com/
# 
#   This program is free software: you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation, version 3 of the License.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see .
# 
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html

# ABOUT:
# - egress traffic (from internal to external):
#     1. is handled directly, applying rules on physical interface (eth*)
#     2. is shaped granularly, using tc classes and filters
# - ingress traffic:
#     1. is handled using ifb interface
#     2. is shaped using tc ingress qdisc (a better classification has to come)

# HOWTO:
# 1) you have ifb interfaces available in your kernel
# 2) every section has its relative howto, see each ones

# Specify parameters of your xDSL. Give values slightly lower than the
# nominal ones. For example, my line is 8000/348, so I specify 8000/320.
# Uplink and downlink speeds
DOWNLINK=8000
UPLINK=320 # not max (348)

# Device that connects you to the Internet
DEV=eth1

# ifb device used to shape ingress traffic
IFB=ifb0

# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIP_IPS="192.168.0.200"
MAX_VOIP_UPLINK=${UPLINK} #max

# PROTO/PORTS SYNTAX - BE CAREFUL
# proto1,proto2:multiple_ports_comma_separated|port-range:[c/s]
# where: proto1,proto2: tcp,udp, and c/s is "client/server"
# i.e. "tcp:24650-24690:c" means "tcp ports from 24650 to 24690 as client"
#
#  "c" and "s" are optionals: 
#  if specified, they are used only, respectively, 
#  "c" for destination traffic (acting as client), 
#  "s" for source (acting as server)
#
#  so, if you have a server listening on tcp 12345 port, use "tcp:12345:s", 
#  while if you have a http server, use "tcp:80,443:cs", since your server 
#  listens on 80 and 443 but your box is also probably a client for external 
#  http server (hence "c")

# VoIP telephony
VOIP_PROTOS="sip rtp rtsp"
VOIP_PORTS="udp:5060,5061 udp:4000-4099:cs udp:16384-16482:cs" # 10000:11000 5000:5059 8000:8016 5004 1720 1731"

# Interactive class: ssh, telnet, dns, vnc, openvpn, irc
INTERACTIVE_PORTS="tcp:22 tcp:23:c udp:53:c tcp:5900,5901 udp:1190 tcp:6667-6670:c"

# WWW, jabber and IRC
BROWSING_PORTS="tcp:80:c tcp:443:c tcp:8080-8082:c"

# The lowest priority traffic: eDonkey, Bittorrent, etc.
P2P_PORTS="tcp,udp:24650-24660 tcp,udp:36650-36660 tcp,udp:46650-46660"
# in the above line we use layer7 filter - be careful it doesn't work if traffic is encrypted
P2P_PROTOS="bittorrent edonkey"
MAX_P2P_UPLINK=$(( ${UPLINK}/10 ))

MTU=1540 # your line mtu

###
### no touch below - or be careful with that axe, eugene!
###

DEBUG=$2

if [ ! -z $DEBUG ] && [ $DEBUG == "demo" ]
then
  IPTABLES="echo iptables"
  TC="echo tc"
else
  IPTABLES="iptables"
  TC="tc"
fi

SHAPER_EGRESS="SHAPER_EGRESS"
IPTABLES_EGRESS_CHAIN="POSTROUTING"

SHAPER_INGRESS="SHAPER_INGRESS"
IPTABLES_INGRESS_CHAIN="PREROUTING"

# Traffic classes:
CLASS_LOW_LATENCY=2 # 1:2 Low latency (VoIP)
CLASS_INTERACTIVE=3 # 1:3 Interactive (SSH, DNS, ACK, OPENVPN)
CLASS_BROWSING=4 # 1:4 Browsing (HTTP, HTTPs)
CLASS_DEFAULT=5 # 1:5 Default
CLASS_LOW_PRIORITY=6 # 1:6 Low priority (p2p, pop3, smtp, etc)

########################################################################
# Configuration ends here
########################################################################

function set_rules_for_downlink() {
  #for now, we use ingress
  local dev=$DEV
  
  # Set up ingress qdisc
  ${TC} qdisc add dev $dev handle ffff: ingress
  
  local port
  local target

  for target in sport dport
  do
    # filter everything related to voip
    for port in 5060 5061 `seq 16384 16390`
    do
     ${TC} filter add dev $dev parent ffff: protocol ip prio 10 \
        u32 match ip src 0.0.0.0/0 \
        match ip protocol 6 0xff \
        match ip $target $port 0xffff \
        police rate $((10*${DOWNLINK}/10))kbit \
        burst 35k flowid :1
    done
  
    # Filter everything that is coming in too fast
    # It's mostly HTTP downloads that keep jamming the downlink, so try to restrict
    # them to 6/10 of the downlink.
    for port in 80 443;
    do
      ${TC} filter add dev $dev parent ffff: protocol ip prio 50 \
 u32 match ip src 0.0.0.0/0 \
 match ip protocol 6 0xff \
 match ip sport $port 0xffff \
 police rate $((6*${DOWNLINK}/10))kbit \
 burst 10k drop flowid :1

      ${TC} filter add dev $dev parent ffff: protocol ip prio 50 \
 u32 match ip src 0.0.0.0/0 \
 match ip protocol 6 0xff \
 match ip dport $port 0xffff \
 police rate $((6*${DOWNLINK}/10))kbit \
 burst 10k drop flowid :1
    done
  done
  
  echo "Ingress shaping applied on $DEV."
}

function set_rules_for_uplink() {
  set_up_rules_for_interface $DEV $UPLINK $MAX_P2P_UPLINK $SHAPER_EGRESS $IPTABLES_EGRESS_CHAIN
  echo "Ingress shaping applied on $DEV."
}

function set_up_rules_for_interface() {
  local dev=$1
  local link=$2
  local max_p2p_link=$3
  local chain=$4
  local iptables_chain=$5
  set_tc_rules_for_interface $dev $link $max_p2p_link
  set_iptables_rules_for_interface $dev $chain $iptables_chain
}

function set_tc_rules_for_interface() {
  local dev=$1      
  local link=$2
  local max_p2p_link=$3

  # add HFSC root qdisc
  ${TC} qdisc add dev $dev root handle 1: hfsc default $CLASS_DEFAULT

  # add main rate limit class
  ${TC} class add dev $dev parent 1: classid 1:1 hfsc \
    sc rate ${link}kbit ul rate ${link}kbit
    
  # VoIP: guarantee full uplink for 200ms, then 10/10 uplink
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_LOW_LATENCY" hfsc \
     sc m1 ${link}kbit d 200ms m2 $((10*$link/10))kbit \
     ul rate ${link}kbit

  # Interactive traffic: guarantee realtime half uplink for 50ms, then
  # 4/10 of the uplink
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_INTERACTIVE" hfsc \
    rt m1   ${link}kbit d  50ms m2 $((5*$link/10))kbit \
    ls m1   ${link}kbit d  50ms m2 $((4*$link/10))kbit \
    ul rate ${link}kbit

  # Browsing: guarantee 7/10 for the first second, then
  # guarantee 1/10
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_BROWSING" hfsc \
    sc m1 $((6*$link/10))kbit d 1s m2 $(($link/10))kbit \
    ul rate ${link}kbit

  # Default traffic: don't guarantee anything for the first two seconds,
  # then guarantee 1/20
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_DEFAULT" hfsc \
    sc m1         0 d    2s m2 $(($link/20))kbit \
    ul rate ${link}kbit

  # low priority traffic: don't guarantee anything for the first 10 seconds,
  # then guarantee 1/30, until it reaches upper limit (uplink/10 as default)
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_LOW_PRIORITY" hfsc \
    ls m1   0 d  10s m2 $(($link/30))kbit \
    ul rate ${max_p2p_link}kbit 

  local class
  for class in $CLASS_LOW_LATENCY $CLASS_INTERACTIVE $CLASS_BROWSING $CLASS_DEFAULT $CLASS_LOW_PRIORITY
  do
    ${TC} filter add dev $dev parent 1: prio $class protocol ip handle $class fw flowid 1:$class
  done
  
  for class in $CLASS_LOW_LATENCY $CLASS_INTERACTIVE $CLASS_BROWSING $CLASS_DEFAULT $CLASS_LOW_PRIORITY
  do
    ${TC} qdisc add dev $dev parent 1:$class handle $class sfq quantum $MTU perturb 10
  done
}

function set_iptables_rules_for_interface() {
  local dev=$1;
  
  local chain=$2
  local chain_pre="${chain}_PRE"
  
  local iptables_chain=$3
  local iptables_chain_pre="${iptables_chain}_PRE"
     
  # add $SHAPER chain to mangle $iptables_chain table
  ${IPTABLES} -t mangle -N $chain
  ${IPTABLES} -t mangle -I $iptables_chain -o $dev  -j $chain

  ${IPTABLES} -t mangle -N $chain_pre
  ${IPTABLES} -t mangle -I $iptables_chain -o $dev -j $chain_pre
  #Restore any previous connection marks
  ${IPTABLES} -t mangle -A $chain_pre -j CONNMARK --restore-mark
  #Do not remark any packets--Accept any packets already marked
  ${IPTABLES} -t mangle -A $chain_pre -m mark ! --mark 0 -j ACCEPT
  
  local -a args;
  local ports_prefs;

  ## FIRST RULES FOR VOIP ##
  
  # first rules by ips
  local voip_ip;  
  for voip_ip in ${VOIP_IPS}
  do
    local -a voip_ips_targets
    if [[ $chain =~ .*EGRESS.* ]]
    then
      voip_ips_targets[0]="--src ${voip_ip} -p udp"
    elif [[ $chain =~ .*INGRESS.* ]]
    then
      voip_ips_targets[1]="--dst ${voip_ip} -p udp"
    fi
    local voip_ips_rule;
    for voip_ips_rule in "${voip_ips_targets[@]}"
    do
      args=("${voip_ips_rule}" "${CLASS_LOW_LATENCY}" "${chain}")
      apply_chain_rules args[@]
    done
  done
  
  # then rules by ports
  #local voip_port;
#  for ports_prefs in $VOIP_PORTS; do set_class_by_port $ports_prefs $CLASS_LOW_LATENCY $chain; done
  set_class_by_ports_prefs "${VOIP_PORTS}" $CLASS_LOW_LATENCY $chain

  # final rules by l7-proto
  local l7_proto_args=("${VOIP_PROTOS}" "${CLASS_LOW_LATENCY}" "$chain")
  apply_class_by_l7_protos l7_proto_args[@]
  #  local voip_proto;
  
  ## SECOND RULES FOR INTERACTIVE TRAFFIC ## 
  
  # ICMP (ip protocol 1) in the interactive class  
  local icmp_rule="-p icmp -m length --length :512"
  args=("${icmp_rule}" "$CLASS_INTERACTIVE" "$chain")
  apply_chain_rules args[@]
  
  # syn to 22 in interactive or browsing class 
  local short_ack_rule="-p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN --dport 22"
  args=("${short_ack_rule}" "$CLASS_INTERACTIVE" "${chain}" "tcp");
  apply_chain_rules args[@]
  
  # remaining rules by ports: 22, 53, 1190, etc
  set_class_by_ports_prefs "${INTERACTIVE_PORTS}" $CLASS_INTERACTIVE $chain


  ## THIRD RULES FOR BROWSING TRAFFIC ##

  # put large (512+) icmp packets in browsing category
  local large_icmp_rule="-p icmp -m length --length 512:"
  args=("${large_icmp_rule}" "$CLASS_BROWSING" "$chain")
  apply_chain_rules args[@]
  
  # syn to 80 in interactive or browsing class 
  local short_ack_rule="-p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN --dport 80"
  args=("${short_ack_rule}" "$CLASS_BROWSING" "${chain}" "tcp");
  apply_chain_rules args[@]
        
  # remaining rules by ports  
  set_class_by_ports_prefs "${BROWSING_PORTS}" $CLASS_BROWSING $chain


  ## FINALLY RULES FOR P2P
  set_class_by_ports_prefs "$P2P_PORTS" $CLASS_LOW_PRIORITY $chain
  
  # final rules by l7-proto
  local l7_proto_args=("${P2P_PROTOS}" "${CLASS_LOW_PRIORITY}" "$chain")
  apply_class_by_l7_protos l7_proto_args[@]
 
  [ ! -z $DEBUG ] && {
    echo debug on
    args=("" $CLASS_DEFAULT $chain)
  }
  
  ${IPTABLES} -t mangle -A $chain_pre -j CONNMARK --save-mark
  ${IPTABLES} -t mangle -A $chain_pre -j RETURN
  
  ${IPTABLES} -t mangle -A $chain -j RETURN
}



# usage: apply_chain_rules (rule,class,chain,[proto]) <- array
function apply_chain_rules() {
  local args=("${!1}")
  local rule=${args[0]}
  local
  local class_mark="${class}"
  local chain=${args[2]}
  local chain_pre="${chain}_PRE"
  local proto=${args[3]}
  
   
  [ $DEBUG ] && echo "rule:${rule} class:${class} chain:$chain chain_pre:$chain_pre proto:$proto"
  ${IPTABLES} -t mangle -A ${chain_pre} \
    ${rule} \
    -j MARK --set-mark ${class_mark}
  if [ ! -z $DEBUG ] 
  then
     ${IPTABLES} -t mangle -A $chain_pre -m mark --mark $class_mark -j LOG --log-prefix "marked with $class_mark -> "
  fi
  ${IPTABLES} -t mangle -A $chain \
    $rule \
    -j CLASSIFY --set-class 1:"$class"
}

function set_class_by_ports_prefs() {
  local ports_prefs=$1
  local
  local chain=$3
   
  local pps;
  local has_range="false"
  for pps in ${ports_prefs}
  do
    local -a prefs_array=(${pps//:/ })
    local -a protos=(${prefs_array[0]//,/ })
    local ports=(${prefs_array[1]//-/:})
    local m_options="";
    [[ $ports =~ .*,.* ]] && has_range="true"
    
    local direction=${prefs_array[2]}
    
    local proto
    local direction_targets
    local direction_target
    for proto in "${protos[@]}"
    do
      if [[ -z $direction ]] || [[ $direction == "cs" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--sports --dports"
 else
   direction_targets="--sport --dport"
 fi
      elif [[ $direction == "s" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--sports"
 else
   direction_targets="--sport"
 fi
      elif [[ $direction == "c" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--dports"
 else
   direction_targets="--dport"
 fi
      fi
      
      for direction_target in $direction_targets
      do 
       local args=($proto "${m_options}" $direction_target $ports $class $chain)
       apply_class_by_port_prefs args[@]
      done
    done    
  done
}

function apply_class_by_port_prefs() {
  local argv=("${!1}")
  local proto=${argv[0]}
  local m_options=${argv[1]}
  local direction=${argv[2]}
  local ports=${argv[3]}
  local
  local shaper_chain=${argv[5]}
  local rule="-p $proto ${m_options} ${direction} ${ports}"
  local args=("${rule}" $class $shaper_chain $proto)
  apply_chain_rules args[@]  
} 

function apply_class_by_port_range() {
  local proto=$1
  local direction=$2
  local port_range=$3
  local
  local shaper_chain=$5
  local rule="-p $proto ${direction} ${port_range}"
  local args=("${rule}" $class $shaper_chain $proto)
  apply_chain_rules args[@]
}

function check_layer7_is_available {
   find /lib/modules/$(uname -r) -iname *xt_layer7.ko* > /dev/null \
      && find /lib/xtables/ -iname *libxt_layer7* > /dev/null \
      && echo 0 \
      || echo 1
}

function apply_class_by_l7_protos__real() {
  local args=("${!1}");
  local l7_protos=${args[0]}
  local class_priority=${args[1]};
  local chain=${args[2]}
  for proto in ${l7_protos[@]}
  do
    local layer7_rule="-m layer7 --l7proto ${proto}"
    local args=("${layer7_rule}" "$class_priority" "$chain")
    apply_chain_rules args[@]
  done  
}
function apply_class_by_l7_protos() {
   local args=("${!1}");
   [ check_layer7_is_available ] && apply_class_by_l7_protos__real args[@]
}

function status() {
  echo ""
  for device in $DEV $IFB;
  do
    if [[ $device =~ "eth" ]]
    then
      echo -e "\tUPLOAD\n"
    else
      echo -e "\tDOWNLOAD\n"
    fi
    
    echo "[qdisc]"
    ${TC} -s qdisc show dev $device

    echo ""
    echo "[class]"
    ${TC} -s class show dev $device

    echo ""
    echo "[filter]"
    ${TC} -s filter show dev $device

    echo ""
    echo "[iptables]"
    if [[ $device =~ "eth" ]]
    then
      ${IPTABLES} -t mangle -L  $SHAPER_EGRESS -v -n 2> /dev/null
      ${IPTABLES} -t mangle -L  "${SHAPER_EGRESS}_PRE" -v -n 2> /dev/null
    else
      ${IPTABLES} -t mangle -L  $SHAPER_INGRESS -v -n 2> /dev/null
      ${IPTABLES} -t mangle -L  "${SHAPER_INPUT}_PRE" -v -n 2> /dev/null
    fi
    
    echo ""
  done
}

function delete_for_interface() {
  local dev=$1
  local chain=$2
  local x_gress=$3
  local x_gress_pre="$3_PRE"
  ${TC} qdisc del dev $dev root  2>/dev/null
  
  # Flush and delete tables
  $IPTABLES -t mangle -D $chain -o $dev -j $x_gress 2>/dev/null
  ${IPTABLES} -t mangle -F $x_gress   2>/dev/null
  ${IPTABLES} -t mangle -X $x_gress  2>/dev/null
  
  ${IPTABLES} -t mangle -D $chain -o $dev -j "${x_gress_pre}" 2>/dev/null
  ${IPTABLES} -t mangle -F "${x_gress_pre}" 2>/dev/null
  ${IPTABLES} -t mangle -X "${x_gress_pre}"  2>/dev/null
}

function delete_for_uplink() {
  delete_for_interface $DEV $IPTABLES_EGRESS_CHAIN $SHAPER_EGRESS
  echo "Egress shaping removed on $DEV."
}

function delete_for_downlink() {
  ${TC} qdisc del dev $DEV ingress > /dev/null 2>&1
  # future:
  # delete_for_interface $IFB_DEV ;
  echo "Ingress shaping removed on $DEV."
}

case "$1" in
status)
  status
  exit
  ;;
stop)
  delete_for_uplink
  delete_for_downlink
  exit
  ;;
start)
  set_rules_for_uplink
  set_rules_for_downlink
  exit
  ;;
restart)
  delete_for_uplink
  delete_for_downlink

  set_rules_for_uplink
  set_rules_for_downlink
  exit
  ;;
*)
  echo "$(basename $0) start|stop|status|restart"
  exit
  ;;
esac


Enjoy it !

May 28, 2013

android emulator x86 with google maps apis

Ok, this night's question is:
can we definitely have an android emulator image using x86 cpu (to exploit kvm acceleration) and google maps support?
The answer is: yes.
But, as always, we could have less pain banging our head against the wall.

However, here the tricks.

  1. Create two avd image, an *arm* ones and an *intel* ones, from shell or using eclipse tools, and assume their names are "ga17_tablet10_arm" and "ga17_tablet10_intel"
    • ga = "google apis"
    • 17 = android selected version
    • tablet10 = a 10'' 1280x800 tablet
    • arm/intel = obviously, isn't?)
    (If *avd" image is already present, skip first creation)
    export ARM_NAME="ga17_tablet10_arm"
    export INTEL_NAME="ga17_tablet10_intel"
    
    or choose different names
  2. Then we have to copy some files from arm instance.
    So, start the avd image and do the stuff [emulator -avd "$ARM_NAME" &]
    Assuming we're working in a dir with enough space (we need about 350mb), we must have a tree as:
    [k0smik0@n-johnny-5-iii from_ga17_arm]$ find system/ -type d
    system/
    system/app
    system/etc
    system/etc/permissions
    system/framework
    
    so
    mkdir -p system/app system/etc/permissions system/framework
  3. Now we have to pull down these files (change ARM_EMU_ID with yours, grabbing it from "adb devices"):
    export ARM_EMU_ID=emulator-5554;
    adb -s $ARM_EMU_ID pull /system/etc/permissions/com.google.android.maps.xml system/etc/permissions
    adb -s $ARM_EMU_ID pull /system/framework/com.google.android.maps.jar system/framework
    adb -s $ARM_EMU_ID pull /system/app/FusedLocation.apk system/app
    adb -s $ARM_EMU_ID pull /system/app/GoogleLoginService.apk system/app
    adb -s $ARM_EMU_ID pull /system/app/GoogleServicesFramework.apk system/app
    adb -s $ARM_EMU_ID pull /system/app/GpsLocationTest.apk system/app
    adb -s $ARM_EMU_ID pull /system/app/Maps.apk system/app
    adb -s $ARM_EMU_ID pull /system/app/NetworkLocation.apk system/app
    
  4. Then, we have to push these files in *x86/intel* avd image, in same directories:
    assuming:
    1. you're using a 64 linux with kvm acceleration (hence "emulator64-x86")
    2. "$INTEL_NAME" is env variable for intel avd image name (see 1, above)
    3. 300 (mb) is enough space you need - and it must be greater than what you have specified on image creation (item 1) [default is 260mb]
    achtung! you have to change data partition size before you start emulator, because new system image will be saved in /data; as you can see u have to increase it:
    grep -ni dataPartition\.size ~/.android/avd/"$INTEL_NAME".avd/*ini
    config.ini:3:disk.dataPartition.size=200M
    hardware-qemu.ini:40:disk.dataPartition.size = 200m 
    
    so change these values in config.ini and hardware-qemu.ini [vim?ed?sed?] from 200M to 400M (or higher, depends how you want expand /system). and now:
    emulator64-x86 -partition-size 300 -netspeed full -netdelay none -no-boot-anim -no-audio -gpu on -avd $INTEL_NAME -qemu -enable-kvm &
    export INTEL_EMU_ID=emulator-5556; # replace with yours
    sleep 10; # adb connection takes some seconds
    adb remount
    adb -s $INTEL_EMU_ID push system/etc/permissions/com.google.android.maps.xml /system/etc/permissions/
    adb -s $INTEL_EMU_ID push system/framework/com.google.android.maps.jar /system/framework/
    for i in system/app/*; do adb -s $INTEL_EMU_ID push $i /system/app ; done
    
  5. Is it enough? No, obviously. These new changes don't remain after avd instance reboot, therefore we have to save /system partition.
    So, save this mkfs.yaffs2.x86 and push it to the device:
    wget https://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86
    adb -s $INTEL_EMU_ID push mkfs.yaffs2.x86 /data
    
  6. create a new system image from existing ones:
    adb -s $INTEL_EMU_ID shell chmod 755 /data/mkfs.yaffs2.x86
    adb -s $INTEL_EMU_ID shell /data/mkfs.yaffs2.x86 /system /data/system.img
    
    and now save new image (this could take a while, so get a beer):
    adb -s $INTEL_EMU_ID pull /data/system.img
    
    just for info: mine reports "829 KB/s (304309632 bytes [~291mb] in 358.296s [~6s])"
    1. ...breathe, it's almost completed.

      Now, choice if u want use new system.img only this specific intel instance, or globally, for new image creations
      * in this second case, in virtual manager ("android avd"), if you select "Google APIs (Google Inc.) - Version XY" in target, you could see a "Intel Atom (x86)" item in cpu/abi, not only "ARM (armeabi-v7a)" *
    2. local/specific instance

      copy system.img in avd directory:
      cp -a system.img ~/.android/avd/"$INTEL_NAME".avd/
      
    3. global method

      copy this new system.img in $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86 - assuming that:
      - ANDROID_SDK is your android directory installation
      - SELECTED_WORKING_VERSION is android release you have choice when you created avd images (I used jellybean_mr1 = 17 -- see item 1)
      and copy other files from x86 (without maps support) sdk dir, that is:
      export ANDROID_SDK=/your/android/installation/dir;
      export SELECTED_WORKING_VERSION=17;
      cp -a $ANDROID_SDK/system-images/android-$SELECTED_WORKING_VERSION/x86/* $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86
      cp -f system.img $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86
      
  7. and finally kill emulators
    adb -s $INTEL_EMU_ID emu kill
    adb -s $ARM_EMU_ID emu kill
    


enjoy this new one intel emulator with google maps ;D




credits to: http://codebutler.com/2012/10/10/configuring-a-usable-android-emulator/

June 24, 2012

Android Compatibility Package and GoogleMaps issues: FragmentMapActivity and MapFragment

Chapter 1: "The Good, The Bad and The Ugly"

A short introduction:

The target - the cemetery treasure:

we want develop an android app for all platforms, from 1.6 to 4.x, having same ui and backend code. i.e., we want write once, and run everywhere.
(Did you know those words, isn't ?)

The requirements - we have to make an alliance with "The Ugly"

So, we have (only?) to use fragment support from compatibility package, in order to take advantage of fragments paradigma in our application, on device which version < 3.x - honeycomb.
Easy, isn't ? Not so much.

The problem - aka "The Bad"

Android compatibility package doesn't offer support for maps in fragments, so there is not mapfragment for us.
Then, we could just implement our own version... just a "MapFragment extends Fragment"; and honoring fragments paradigma, we let fragment contains all ui/event code.
And an obvious thing is to be able to instance MapView in our MapFragment.

But immediately the first problem: we know a *Fragment must be attached to *FragmentActivity, and we know a MapView must living in a MapActivity - and we don't have a MapActivity with fragment support. There is nothing about this in compatibility package.
The unique existing FragmentActivity inherits from Activity, so the problem.

The road to solution, "The Good"

Pete Doyle proposes first solution, modifiyng the compatibility package source, and we can have a FragmentActivity inheriting from MapActivity.
You can see all about it on his github https://github.com/petedoyle/android-support-v4-googlemaps.
However, the problem is that only one MapActivity is supported per process (here details)
Taro Kobayashi talks about this in his blog http://uwvm.blogspot.jp
So, two alternatives:
  1. we have only and only one FragmentActivity inheriting from MapActivity in our app, and we have to project ui as a "detach current fragment" then "attach next fragment" ("replace" method has some issues in fragment support)
  2. have a distinct FragmentMapActivity inheriting from MapActivity, and obviously providing fragment support while honors compatibility package.
    reading in Kobayashi blog, it's evident he prefers this second way.. et voila his fork from petedoyle's project:
    https://github.com/9re/android-support-v4-googlemaps

And yes, I also searched a solution for all this issue, and even providing the second way too.
Searched, tried to implement my ones, and so on. As long as i just found Kobayashi ones ;D So, it just remains to use ? Oh no, nothing is as easy as it sounds.

"Hey Blondie! You know what you are? Just a dirty son of a ahAHahAHaaaaah!"

As already shown in petedoyle's project, there is some issue with MapFragment, especially instancing MapView:
https://github.com/petedoyle/android-support-v4-googlemaps/blob/master/samples/FragmentLayoutMaps/src/com/example/android/apis/app/FragmentLayout.java plz focus on "onCreateView", row 224 (and row 80, for fields declarations)


As you could see, Pete instances a View inflating from R.layout.map, that is file containing <com.google.android.maps.MapView> (or another MapView), and then this view is returned as viewcontainer in onCreateView.
In onCreateView body he use mapview and does his stuffs, and you can see as View mapView and View mapViewContainer are instanced in parent FragmentMapActivity. But why ?

Fragment borns to contains all ui and event code, and handles fields life within its lifecycle, so we would instance our fields in our MapFragment, not in FragmentMapActivity.

Then, my endeavour (anyhow, it works) for MapFragment:
  • it handles viewContainer directly, instancing it in onAttach;
  • it is an abstract class, with "providesMapViewContainerId" abstract method;
    in concrete subclass, it provides file layout resource id
  • a concrete (final?) "addMapViewToViewParent" method, to be invoked within concrete subclass onCreateView, passing View parameter as its first parameter, and our MapView as second ones.


And MapView where it comes from?
Simply with findViewById (or injecting as singleton, if you use roboguice). So:
public MyMapFragment extends MapFragment {
   private MapView mapView;
   /*
    * other methods
    */
   @Override
   public void onViewCreated(View view, Bundle savedInstanceState) {
      super.onViewCreated(view, savedInstanceState);
      if (mapView == null) // cause only mapview is admitted
         mapView = getActivity().findViewById(R.id.yourmapview);
      addMapViewToViewParent(view,mapView);

      // your stuffs with mapview
   }
}

and MapFragment.java on github or code below:
/*
 * Copyright (C) 2012 Massimiliano Leone - maximilianus@gmail.com
 * Licensed under the Apache License, Version 2.0 (the "License");
 */
package android.support.v4.app;

import android.app.Activity;
import android.os.Bundle;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.FrameLayout;

import com.google.android.maps.MapView;

public abstract class MapFragment extends Fragment {

  private View mapViewContainer;

  /**
    * inflating from layout provided by (your concrete) providesMapViewContainerId method;
    * here we instance the viewcontainer returned by onViewCreated,
    * something as:
    *
    * <pre>return LayoutInflater.from(activity).inflate(R.layout.yourmaplayout, null);* </pre>
    */
  @Override
  public void onAttach(Activity activity) {
    super.onAttach(activity);       

    if (mapViewContainer == null) {
      mapViewContainer =  LayoutInflater.from(activity).inflate( providesMapViewContainerId(), null);
    }
  }

  /**
    * in your concrete class you have to provide the layout containing mapview:
    * something as:
    *
    * "return R.layout.map;"
    * where R.layout.map is map.xml file in layout directory, declaring a RelativeLayout,
    * which contains Mapview, as below:
    *
    * <pre>
    *  <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    *   android:id="@+id/maplayout"
    *   android:orientation="vertical"
    *   android:layout_width="fill_parent"
    *   android:layout_height="fill_parent">

    *  <com.google.android.maps.MapView
    *    android:id="@+id/mapview"
    *    android:layout_width="fill_parent"
    *    android:layout_height="fill_parent"
    *    android:clickable="true"
    *    android:apiKey="YOUR_APIKEY"/>
    *  </RelativeLayout>
    * </pre>    
    * @param activity
    * @return View
    */
  protected abstract int providesMapViewContainerId();

  protected void addMapViewToViewParent(View view, MapView mapView) {

    ViewGroup viewGroup = (ViewGroup) view.getParent();
    if (viewGroup == null) viewGroup = (ViewGroup) view;
    int childs = viewGroup.getChildCount();
    if (childs == 0) {
 viewGroup.addView(mapView);
    } else {
 for (int i = 0; i < viewGroup.getChildCount(); i++) {
   View child = viewGroup.getChildAt(i);
   if (child instanceof FrameLayout) {
       continue;
   } else if (child instanceof MapView) {
       viewGroup.removeView(child);
       viewGroup.addView(mapView, 1);
   } else {
       viewGroup.removeView(child);
   }
      }
      //viewGroup.addView(mapView, 0);
    }
    viewGroup.invalidate();
  }   

  @Override
  public View onCreateView(LayoutInflater inflater, ViewGroup container,Bundle savedInstanceState) {
  return mapViewContainer;
  }

  @Override
  public void onDestroyView() {
    super.onDestroyView();
    ViewGroup parentViewGroup = (ViewGroup) mapViewContainer.getParent();
    if( parentViewGroup != null ) {
      parentViewGroup.removeView( mapViewContainer );
      //parentViewGroup.removeAllViews();
    }
  }   

}