openfarmacie@ravenna is out

OpenFarmacie@Ravenna is out!

http://openfarmacieravenna.url.ph

2^ place at www.ra4open.it
Award party at #sod14 SpaghettiOpenData, the italian opendata folks in Bologna.

Enjoy it!

ehfsc - the (almost) ultimate solution for traffic shaping in linux

E-HSFC a.k.a. Enhanced HFSC

Some people know hfsc, an efficient traffic classifier available in Linux kernel.
For who doesn't, it's time to do ;D

http://en.wikipedia.org/wiki/Hierarchical_fair-service_curve
and
http://www.cs.cmu.edu/~hzhang/HFSC/main.html

In 2 words: it allows link-sharing, real-time, adaptive best-effort in a hierarchical way, better than HTB or other hierarchical classifiers.

HFSC is activable via "tc" tools, well-known for its hard syntax, even when used in conjunction with iptables.


So, e-hfsc is a powerful script for people who don't want wast time in annoying stuff like tc commands:
it's just a "2-minutes config-and-start".

There are some mandatory values to set, as, i.e.:

DOWNLINK=8000 
UPLINK=320 
# Device that connects you to the Internet 
DEV=eth1
#
# other values to set, see file header
#

or, for ports you want apply shaping on, something as:

INTERACTIVE_PORTS="tcp:22 tcp:23:c udp:53:c tcp:5900,5901 udp:1190 tcp:6667-6670:c"

# where the syntax is:
# proto1,proto2:multiple_ports_comma_separated|port-range:[c/s]
# where: proto1,proto2: tcp,udp, and c/s is "client/server"
# i.e. "tcp:24650-24690:c" means "tcp ports from 24650 to 24690 as client"

# "c" and "s" are optionals: if specified, they are used only, respectively, 
# "c" for destination traffic (acting as client), 
# "s" for source (acting as server)

# so, if you have a server listening on tcp 12345 port, use "tcp:12345:s", 
# while if you have a http server, use "tcp:80,443:cs", since your server 
# surely listens on 80 and 443, but your box is also probably a client 
# for external http server (hence "c").

There are other filters available, that is "IP" and "L7-PROTO":
IP is easy to understand (I hope ;D), while "L7-PROTO" refers to same name netfilter/iptables module,
"l7-proto", that alllows to match traffic type specifying its name according to ISO-OSI layer 7 rfc,
i.e. "ssh", "ftp", "http", "torrent", and so on.
Since it comes with kernel modules, if its relative .ko file and iptables .so file could be not retrieved,
e-hfsc will simply ignore rules using l7-proto.

All (mandatory and optional) values must be setted in file header.


Finally:

./e-hfsc start|stop|restart

Check last version (and fork) at https://github.com/k0smik0/e-hfsc !

The image above show last 24h statistics of my network:

For people who want directly read the code, voilà:

#!/bin/bash
# encoding: UTF-8
#
# E-HFSC ("enhanced-hfsc") is a script for make better gain for voip, interactive and http 
# traffic, allowing even the p2p one
#
# heavily based on Bliziński's hfsc script, from http://automatthias.wordpress.com/
# 
#   This program is free software: you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation, version 3 of the License.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see .
# 
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html

# ABOUT:
# - egress traffic (from internal to external):
#     1. is handled directly, applying rules on physical interface (eth*)
#     2. is shaped granularly, using tc classes and filters
# - ingress traffic:
#     1. is handled using ifb interface
#     2. is shaped using tc ingress qdisc (a better classification has to come)

# HOWTO:
# 1) you have ifb interfaces available in your kernel
# 2) every section has its relative howto, see each ones

# Specify parameters of your xDSL. Give values slightly lower than the
# nominal ones. For example, my line is 8000/348, so I specify 8000/320.
# Uplink and downlink speeds
DOWNLINK=8000
UPLINK=320 # not max (348)

# Device that connects you to the Internet
DEV=eth1

# ifb device used to shape ingress traffic
IFB=ifb0

# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIP_IPS="192.168.0.200"
MAX_VOIP_UPLINK=${UPLINK} #max

# PROTO/PORTS SYNTAX - BE CAREFUL
# proto1,proto2:multiple_ports_comma_separated|port-range:[c/s]
# where: proto1,proto2: tcp,udp, and c/s is "client/server"
# i.e. "tcp:24650-24690:c" means "tcp ports from 24650 to 24690 as client"
#
#  "c" and "s" are optionals: 
#  if specified, they are used only, respectively, 
#  "c" for destination traffic (acting as client), 
#  "s" for source (acting as server)
#
#  so, if you have a server listening on tcp 12345 port, use "tcp:12345:s", 
#  while if you have a http server, use "tcp:80,443:cs", since your server 
#  listens on 80 and 443 but your box is also probably a client for external 
#  http server (hence "c")

# VoIP telephony
VOIP_PROTOS="sip rtp rtsp"
VOIP_PORTS="udp:5060,5061 udp:4000-4099:cs udp:16384-16482:cs" # 10000:11000 5000:5059 8000:8016 5004 1720 1731"

# Interactive class: ssh, telnet, dns, vnc, openvpn, irc
INTERACTIVE_PORTS="tcp:22 tcp:23:c udp:53:c tcp:5900,5901 udp:1190 tcp:6667-6670:c"

# WWW, jabber and IRC
BROWSING_PORTS="tcp:80:c tcp:443:c tcp:8080-8082:c"

# The lowest priority traffic: eDonkey, Bittorrent, etc.
P2P_PORTS="tcp,udp:24650-24660 tcp,udp:36650-36660 tcp,udp:46650-46660"
# in the above line we use layer7 filter - be careful it doesn't work if traffic is encrypted
P2P_PROTOS="bittorrent edonkey"
MAX_P2P_UPLINK=$(( ${UPLINK}/10 ))

MTU=1540 # your line mtu

###
### no touch below - or be careful with that axe, eugene!
###

DEBUG=$2

if [ ! -z $DEBUG ] && [ $DEBUG == "demo" ]
then
  IPTABLES="echo iptables"
  TC="echo tc"
else
  IPTABLES="iptables"
  TC="tc"
fi

SHAPER_EGRESS="SHAPER_EGRESS"
IPTABLES_EGRESS_CHAIN="POSTROUTING"

SHAPER_INGRESS="SHAPER_INGRESS"
IPTABLES_INGRESS_CHAIN="PREROUTING"

# Traffic classes:
CLASS_LOW_LATENCY=2 # 1:2 Low latency (VoIP)
CLASS_INTERACTIVE=3 # 1:3 Interactive (SSH, DNS, ACK, OPENVPN)
CLASS_BROWSING=4 # 1:4 Browsing (HTTP, HTTPs)
CLASS_DEFAULT=5 # 1:5 Default
CLASS_LOW_PRIORITY=6 # 1:6 Low priority (p2p, pop3, smtp, etc)

########################################################################
# Configuration ends here
########################################################################

function set_rules_for_downlink() {
  #for now, we use ingress
  local dev=$DEV
  
  # Set up ingress qdisc
  ${TC} qdisc add dev $dev handle ffff: ingress
  
  local port
  local target

  for target in sport dport
  do
    # filter everything related to voip
    for port in 5060 5061 `seq 16384 16390`
    do
     ${TC} filter add dev $dev parent ffff: protocol ip prio 10 \
        u32 match ip src 0.0.0.0/0 \
        match ip protocol 6 0xff \
        match ip $target $port 0xffff \
        police rate $((10*${DOWNLINK}/10))kbit \
        burst 35k flowid :1
    done
  
    # Filter everything that is coming in too fast
    # It's mostly HTTP downloads that keep jamming the downlink, so try to restrict
    # them to 6/10 of the downlink.
    for port in 80 443;
    do
      ${TC} filter add dev $dev parent ffff: protocol ip prio 50 \
 u32 match ip src 0.0.0.0/0 \
 match ip protocol 6 0xff \
 match ip sport $port 0xffff \
 police rate $((6*${DOWNLINK}/10))kbit \
 burst 10k drop flowid :1

      ${TC} filter add dev $dev parent ffff: protocol ip prio 50 \
 u32 match ip src 0.0.0.0/0 \
 match ip protocol 6 0xff \
 match ip dport $port 0xffff \
 police rate $((6*${DOWNLINK}/10))kbit \
 burst 10k drop flowid :1
    done
  done
  
  echo "Ingress shaping applied on $DEV."
}

function set_rules_for_uplink() {
  set_up_rules_for_interface $DEV $UPLINK $MAX_P2P_UPLINK $SHAPER_EGRESS $IPTABLES_EGRESS_CHAIN
  echo "Ingress shaping applied on $DEV."
}

function set_up_rules_for_interface() {
  local dev=$1
  local link=$2
  local max_p2p_link=$3
  local chain=$4
  local iptables_chain=$5
  set_tc_rules_for_interface $dev $link $max_p2p_link
  set_iptables_rules_for_interface $dev $chain $iptables_chain
}

function set_tc_rules_for_interface() {
  local dev=$1      
  local link=$2
  local max_p2p_link=$3

  # add HFSC root qdisc
  ${TC} qdisc add dev $dev root handle 1: hfsc default $CLASS_DEFAULT

  # add main rate limit class
  ${TC} class add dev $dev parent 1: classid 1:1 hfsc \
    sc rate ${link}kbit ul rate ${link}kbit
    
  # VoIP: guarantee full uplink for 200ms, then 10/10 uplink
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_LOW_LATENCY" hfsc \
     sc m1 ${link}kbit d 200ms m2 $((10*$link/10))kbit \
     ul rate ${link}kbit

  # Interactive traffic: guarantee realtime half uplink for 50ms, then
  # 4/10 of the uplink
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_INTERACTIVE" hfsc \
    rt m1   ${link}kbit d  50ms m2 $((5*$link/10))kbit \
    ls m1   ${link}kbit d  50ms m2 $((4*$link/10))kbit \
    ul rate ${link}kbit

  # Browsing: guarantee 7/10 for the first second, then
  # guarantee 1/10
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_BROWSING" hfsc \
    sc m1 $((6*$link/10))kbit d 1s m2 $(($link/10))kbit \
    ul rate ${link}kbit

  # Default traffic: don't guarantee anything for the first two seconds,
  # then guarantee 1/20
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_DEFAULT" hfsc \
    sc m1         0 d    2s m2 $(($link/20))kbit \
    ul rate ${link}kbit

  # low priority traffic: don't guarantee anything for the first 10 seconds,
  # then guarantee 1/30, until it reaches upper limit (uplink/10 as default)
  ${TC} class add dev $dev parent 1:1  classid 1:"$CLASS_LOW_PRIORITY" hfsc \
    ls m1   0 d  10s m2 $(($link/30))kbit \
    ul rate ${max_p2p_link}kbit 

  local class
  for class in $CLASS_LOW_LATENCY $CLASS_INTERACTIVE $CLASS_BROWSING $CLASS_DEFAULT $CLASS_LOW_PRIORITY
  do
    ${TC} filter add dev $dev parent 1: prio $class protocol ip handle $class fw flowid 1:$class
  done
  
  for class in $CLASS_LOW_LATENCY $CLASS_INTERACTIVE $CLASS_BROWSING $CLASS_DEFAULT $CLASS_LOW_PRIORITY
  do
    ${TC} qdisc add dev $dev parent 1:$class handle $class sfq quantum $MTU perturb 10
  done
}

function set_iptables_rules_for_interface() {
  local dev=$1;
  
  local chain=$2
  local chain_pre="${chain}_PRE"
  
  local iptables_chain=$3
  local iptables_chain_pre="${iptables_chain}_PRE"
     
  # add $SHAPER chain to mangle $iptables_chain table
  ${IPTABLES} -t mangle -N $chain
  ${IPTABLES} -t mangle -I $iptables_chain -o $dev  -j $chain

  ${IPTABLES} -t mangle -N $chain_pre
  ${IPTABLES} -t mangle -I $iptables_chain -o $dev -j $chain_pre
  #Restore any previous connection marks
  ${IPTABLES} -t mangle -A $chain_pre -j CONNMARK --restore-mark
  #Do not remark any packets--Accept any packets already marked
  ${IPTABLES} -t mangle -A $chain_pre -m mark ! --mark 0 -j ACCEPT
  
  local -a args;
  local ports_prefs;

  ## FIRST RULES FOR VOIP ##
  
  # first rules by ips
  local voip_ip;  
  for voip_ip in ${VOIP_IPS}
  do
    local -a voip_ips_targets
    if [[ $chain =~ .*EGRESS.* ]]
    then
      voip_ips_targets[0]="--src ${voip_ip} -p udp"
    elif [[ $chain =~ .*INGRESS.* ]]
    then
      voip_ips_targets[1]="--dst ${voip_ip} -p udp"
    fi
    local voip_ips_rule;
    for voip_ips_rule in "${voip_ips_targets[@]}"
    do
      args=("${voip_ips_rule}" "${CLASS_LOW_LATENCY}" "${chain}")
      apply_chain_rules args[@]
    done
  done
  
  # then rules by ports
  #local voip_port;
#  for ports_prefs in $VOIP_PORTS; do set_class_by_port $ports_prefs $CLASS_LOW_LATENCY $chain; done
  set_class_by_ports_prefs "${VOIP_PORTS}" $CLASS_LOW_LATENCY $chain

  # final rules by l7-proto
  local l7_proto_args=("${VOIP_PROTOS}" "${CLASS_LOW_LATENCY}" "$chain")
  apply_class_by_l7_protos l7_proto_args[@]
  #  local voip_proto;
  
  ## SECOND RULES FOR INTERACTIVE TRAFFIC ## 
  
  # ICMP (ip protocol 1) in the interactive class  
  local icmp_rule="-p icmp -m length --length :512"
  args=("${icmp_rule}" "$CLASS_INTERACTIVE" "$chain")
  apply_chain_rules args[@]
  
  # syn to 22 in interactive or browsing class 
  local short_ack_rule="-p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN --dport 22"
  args=("${short_ack_rule}" "$CLASS_INTERACTIVE" "${chain}" "tcp");
  apply_chain_rules args[@]
  
  # remaining rules by ports: 22, 53, 1190, etc
  set_class_by_ports_prefs "${INTERACTIVE_PORTS}" $CLASS_INTERACTIVE $chain


  ## THIRD RULES FOR BROWSING TRAFFIC ##

  # put large (512+) icmp packets in browsing category
  local large_icmp_rule="-p icmp -m length --length 512:"
  args=("${large_icmp_rule}" "$CLASS_BROWSING" "$chain")
  apply_chain_rules args[@]
  
  # syn to 80 in interactive or browsing class 
  local short_ack_rule="-p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN --dport 80"
  args=("${short_ack_rule}" "$CLASS_BROWSING" "${chain}" "tcp");
  apply_chain_rules args[@]
        
  # remaining rules by ports  
  set_class_by_ports_prefs "${BROWSING_PORTS}" $CLASS_BROWSING $chain


  ## FINALLY RULES FOR P2P
  set_class_by_ports_prefs "$P2P_PORTS" $CLASS_LOW_PRIORITY $chain
  
  # final rules by l7-proto
  local l7_proto_args=("${P2P_PROTOS}" "${CLASS_LOW_PRIORITY}" "$chain")
  apply_class_by_l7_protos l7_proto_args[@]
 
  [ ! -z $DEBUG ] && {
    echo debug on
    args=("" $CLASS_DEFAULT $chain)
  }
  
  ${IPTABLES} -t mangle -A $chain_pre -j CONNMARK --save-mark
  ${IPTABLES} -t mangle -A $chain_pre -j RETURN
  
  ${IPTABLES} -t mangle -A $chain -j RETURN
}



# usage: apply_chain_rules (rule,class,chain,[proto]) <- array
function apply_chain_rules() {
  local args=("${!1}")
  local rule=${args[0]}
  local
  local class_mark="${class}"
  local chain=${args[2]}
  local chain_pre="${chain}_PRE"
  local proto=${args[3]}
  
   
  [ $DEBUG ] && echo "rule:${rule} class:${class} chain:$chain chain_pre:$chain_pre proto:$proto"
  ${IPTABLES} -t mangle -A ${chain_pre} \
    ${rule} \
    -j MARK --set-mark ${class_mark}
  if [ ! -z $DEBUG ] 
  then
     ${IPTABLES} -t mangle -A $chain_pre -m mark --mark $class_mark -j LOG --log-prefix "marked with $class_mark -> "
  fi
  ${IPTABLES} -t mangle -A $chain \
    $rule \
    -j CLASSIFY --set-class 1:"$class"
}

function set_class_by_ports_prefs() {
  local ports_prefs=$1
  local
  local chain=$3
   
  local pps;
  local has_range="false"
  for pps in ${ports_prefs}
  do
    local -a prefs_array=(${pps//:/ })
    local -a protos=(${prefs_array[0]//,/ })
    local ports=(${prefs_array[1]//-/:})
    local m_options="";
    [[ $ports =~ .*,.* ]] && has_range="true"
    
    local direction=${prefs_array[2]}
    
    local proto
    local direction_targets
    local direction_target
    for proto in "${protos[@]}"
    do
      if [[ -z $direction ]] || [[ $direction == "cs" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--sports --dports"
 else
   direction_targets="--sport --dport"
 fi
      elif [[ $direction == "s" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--sports"
 else
   direction_targets="--sport"
 fi
      elif [[ $direction == "c" ]]; then
 if [[ $has_range == "true" ]]; then
   m_options="-m multiport"
   direction_targets="--dports"
 else
   direction_targets="--dport"
 fi
      fi
      
      for direction_target in $direction_targets
      do 
       local args=($proto "${m_options}" $direction_target $ports $class $chain)
       apply_class_by_port_prefs args[@]
      done
    done    
  done
}

function apply_class_by_port_prefs() {
  local argv=("${!1}")
  local proto=${argv[0]}
  local m_options=${argv[1]}
  local direction=${argv[2]}
  local ports=${argv[3]}
  local
  local shaper_chain=${argv[5]}
  local rule="-p $proto ${m_options} ${direction} ${ports}"
  local args=("${rule}" $class $shaper_chain $proto)
  apply_chain_rules args[@]  
} 

function apply_class_by_port_range() {
  local proto=$1
  local direction=$2
  local port_range=$3
  local
  local shaper_chain=$5
  local rule="-p $proto ${direction} ${port_range}"
  local args=("${rule}" $class $shaper_chain $proto)
  apply_chain_rules args[@]
}

function check_layer7_is_available {
   find /lib/modules/$(uname -r) -iname *xt_layer7.ko* > /dev/null \
      && find /lib/xtables/ -iname *libxt_layer7* > /dev/null \
      && echo 0 \
      || echo 1
}

function apply_class_by_l7_protos__real() {
  local args=("${!1}");
  local l7_protos=${args[0]}
  local class_priority=${args[1]};
  local chain=${args[2]}
  for proto in ${l7_protos[@]}
  do
    local layer7_rule="-m layer7 --l7proto ${proto}"
    local args=("${layer7_rule}" "$class_priority" "$chain")
    apply_chain_rules args[@]
  done  
}
function apply_class_by_l7_protos() {
   local args=("${!1}");
   [ check_layer7_is_available ] && apply_class_by_l7_protos__real args[@]
}

function status() {
  echo ""
  for device in $DEV $IFB;
  do
    if [[ $device =~ "eth" ]]
    then
      echo -e "\tUPLOAD\n"
    else
      echo -e "\tDOWNLOAD\n"
    fi
    
    echo "[qdisc]"
    ${TC} -s qdisc show dev $device

    echo ""
    echo "[class]"
    ${TC} -s class show dev $device

    echo ""
    echo "[filter]"
    ${TC} -s filter show dev $device

    echo ""
    echo "[iptables]"
    if [[ $device =~ "eth" ]]
    then
      ${IPTABLES} -t mangle -L  $SHAPER_EGRESS -v -n 2> /dev/null
      ${IPTABLES} -t mangle -L  "${SHAPER_EGRESS}_PRE" -v -n 2> /dev/null
    else
      ${IPTABLES} -t mangle -L  $SHAPER_INGRESS -v -n 2> /dev/null
      ${IPTABLES} -t mangle -L  "${SHAPER_INPUT}_PRE" -v -n 2> /dev/null
    fi
    
    echo ""
  done
}

function delete_for_interface() {
  local dev=$1
  local chain=$2
  local x_gress=$3
  local x_gress_pre="$3_PRE"
  ${TC} qdisc del dev $dev root  2>/dev/null
  
  # Flush and delete tables
  $IPTABLES -t mangle -D $chain -o $dev -j $x_gress 2>/dev/null
  ${IPTABLES} -t mangle -F $x_gress   2>/dev/null
  ${IPTABLES} -t mangle -X $x_gress  2>/dev/null
  
  ${IPTABLES} -t mangle -D $chain -o $dev -j "${x_gress_pre}" 2>/dev/null
  ${IPTABLES} -t mangle -F "${x_gress_pre}" 2>/dev/null
  ${IPTABLES} -t mangle -X "${x_gress_pre}"  2>/dev/null
}

function delete_for_uplink() {
  delete_for_interface $DEV $IPTABLES_EGRESS_CHAIN $SHAPER_EGRESS
  echo "Egress shaping removed on $DEV."
}

function delete_for_downlink() {
  ${TC} qdisc del dev $DEV ingress > /dev/null 2>&1
  # future:
  # delete_for_interface $IFB_DEV ;
  echo "Ingress shaping removed on $DEV."
}

case "$1" in
status)
  status
  exit
  ;;
stop)
  delete_for_uplink
  delete_for_downlink
  exit
  ;;
start)
  set_rules_for_uplink
  set_rules_for_downlink
  exit
  ;;
restart)
  delete_for_uplink
  delete_for_downlink

  set_rules_for_uplink
  set_rules_for_downlink
  exit
  ;;
*)
  echo "$(basename $0) start|stop|status|restart"
  exit
  ;;
esac

Enjoy it !

android emulator x86 with google maps apis

Ok, this night's question is:
can we definitely have an android emulator image using x86 cpu (to exploit kvm acceleration) and google maps support?
The answer is: yes.
But, as always, we could have less pain banging our head against the wall.

However, here the tricks.

1. Create two avd image, an *arm* ones and an *intel* ones, from shell or using eclipse tools, and assume their names are "ga17_tablet10_arm" and "ga17_tablet10_intel"
   • ga = "google apis"
   • 17 = android selected version
   • tablet10 = a 10'' 1280x800 tablet
   • arm/intel = obviously, isn't?)
   (If *avd" image is already present, skip first creation)

   export ARM_NAME="ga17_tablet10_arm"
   export INTEL_NAME="ga17_tablet10_intel"
   

   or choose different names
2. Then we have to copy some files from arm instance.
   So, start the avd image and do the stuff [emulator -avd "$ARM_NAME" &]
   Assuming we're working in a dir with enough space (we need about 350mb), we must have a tree as:

   [k0smik0@n-johnny-5-iii from_ga17_arm]$ find system/ -type d
   system/
   system/app
   system/etc
   system/etc/permissions
   system/framework
   

   so

   mkdir -p system/app system/etc/permissions system/framework

3. Now we have to pull down these files (change ARM_EMU_ID with yours, grabbing it from "adb devices"):

   export ARM_EMU_ID=emulator-5554;
   adb -s $ARM_EMU_ID pull /system/etc/permissions/com.google.android.maps.xml system/etc/permissions
   adb -s $ARM_EMU_ID pull /system/framework/com.google.android.maps.jar system/framework
   adb -s $ARM_EMU_ID pull /system/app/FusedLocation.apk system/app
   adb -s $ARM_EMU_ID pull /system/app/GoogleLoginService.apk system/app
   adb -s $ARM_EMU_ID pull /system/app/GoogleServicesFramework.apk system/app
   adb -s $ARM_EMU_ID pull /system/app/GpsLocationTest.apk system/app
   adb -s $ARM_EMU_ID pull /system/app/Maps.apk system/app
   adb -s $ARM_EMU_ID pull /system/app/NetworkLocation.apk system/app
   

4. Then, we have to push these files in *x86/intel* avd image, in same directories:
   assuming:
   1. you're using a 64 linux with kvm acceleration (hence "emulator64-x86")
   2. "$INTEL_NAME" is env variable for intel avd image name (see 1, above)
   3. 300 (mb) is enough space you need - and it must be greater than what you have specified on image creation (item 1) [default is 260mb]
   achtung! you have to change data partition size before you start emulator, because new system image will be saved in /data; as you can see u have to increase it:

   grep -ni dataPartition\.size ~/.android/avd/"$INTEL_NAME".avd/*ini
   config.ini:3:disk.dataPartition.size=200M
   hardware-qemu.ini:40:disk.dataPartition.size = 200m 
   

   so change these values in config.ini and hardware-qemu.ini [vim?ed?sed?] from 200M to 400M (or higher, depends how you want expand /system). and now:

   emulator64-x86 -partition-size 300 -netspeed full -netdelay none -no-boot-anim -no-audio -gpu on -avd $INTEL_NAME -qemu -enable-kvm &
   export INTEL_EMU_ID=emulator-5556; # replace with yours
   sleep 10; # adb connection takes some seconds
   adb remount
   adb -s $INTEL_EMU_ID push system/etc/permissions/com.google.android.maps.xml /system/etc/permissions/
   adb -s $INTEL_EMU_ID push system/framework/com.google.android.maps.jar /system/framework/
   for i in system/app/*; do adb -s $INTEL_EMU_ID push $i /system/app ; done
   

5. Is it enough? No, obviously. These new changes don't remain after avd instance reboot, therefore we have to save /system partition.
   So, save this mkfs.yaffs2.x86 and push it to the device:

   wget https://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86
   adb -s $INTEL_EMU_ID push mkfs.yaffs2.x86 /data
   

6. create a new system image from existing ones:

   adb -s $INTEL_EMU_ID shell chmod 755 /data/mkfs.yaffs2.x86
   adb -s $INTEL_EMU_ID shell /data/mkfs.yaffs2.x86 /system /data/system.img
   

   and now save new image (this could take a while, so get a beer):

   adb -s $INTEL_EMU_ID pull /data/system.img
   

   just for info: mine reports "829 KB/s (304309632 bytes [~291mb] in 358.296s [~6s])"
7. ...breathe, it's almost completed.
   
   Now, choice if u want use new system.img only this specific intel instance, or globally, for new image creations
   * in this second case, in virtual manager ("android avd"), if you select "Google APIs (Google Inc.) - Version XY" in target, you could see a "Intel Atom (x86)" item in cpu/abi, not only "ARM (armeabi-v7a)" *
   1. local/specific instance
      
      copy system.img in avd directory:

      cp -a system.img ~/.android/avd/"$INTEL_NAME".avd/
      

   2. global method
      
      copy this new system.img in $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86 - assuming that:
      - ANDROID_SDK is your android directory installation
      - SELECTED_WORKING_VERSION is android release you have choice when you created avd images (I used jellybean_mr1 = 17 -- see item 1)
      and copy other files from x86 (without maps support) sdk dir, that is:

      export ANDROID_SDK=/your/android/installation/dir;
      export SELECTED_WORKING_VERSION=17;
      cp -a $ANDROID_SDK/system-images/android-$SELECTED_WORKING_VERSION/x86/* $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86
      cp -f system.img $ANDROID_SDK/add-ons/addon-google_apis-google-$SELECTED_WORKING_VERSION/images/x86
      

8. and finally kill emulators

   adb -s $INTEL_EMU_ID emu kill
   adb -s $ARM_EMU_ID emu kill
   

enjoy this new one intel emulator with google maps ;D




credits to: http://codebutler.com/2012/10/10/configuring-a-usable-android-emulator/

Android Compatibility Package and GoogleMaps issues: FragmentMapActivity and MapFragmentChapter 1: "The Good, The Bad and The Ugly"

A short introduction:

The target - the cemetery treasure:

we want develop an android app for all platforms, from 1.6 to 4.x, having same ui and backend code. i.e., we want write once, and run everywhere.
(Did you know those words, isn't ?)

The requirements - we have to make an alliance with "The Ugly"

So, we have (only?) to use fragment support from compatibility package, in order to take advantage of fragments paradigma in our application, on device which version < 3.x - honeycomb.
Easy, isn't ? Not so much.

The problem - aka "The Bad"

Android compatibility package doesn't offer support for maps in fragments, so there is not mapfragment for us.
Then, we could just implement our own version... just a "MapFragment extends Fragment"; and honoring fragments paradigma, we let fragment contains all ui/event code.
And an obvious thing is to be able to instance MapView in our MapFragment.

But immediately the first problem: we know a *Fragment must be attached to *FragmentActivity, and we know a MapView must living in a MapActivity - and we don't have a MapActivity with fragment support. There is nothing about this in compatibility package.
The unique existing FragmentActivity inherits from Activity, so the problem.

The road to solution, "The Good"

Pete Doyle proposes first solution, modifiyng the compatibility package source, and we can have a FragmentActivity inheriting from MapActivity.
You can see all about it on his github https://github.com/petedoyle/android-support-v4-googlemaps.
However, the problem is that only one MapActivity is supported per process (here details)
Taro Kobayashi talks about this in his blog http://uwvm.blogspot.jp
So, two alternatives:

1. we have only and only one FragmentActivity inheriting from MapActivity in our app, and we have to project ui as a "detach current fragment" then "attach next fragment" ("replace" method has some issues in fragment support)
2. have a distinct FragmentMapActivity inheriting from MapActivity, and obviously providing fragment support while honors compatibility package.
   reading in Kobayashi blog, it's evident he prefers this second way.. et voila his fork from petedoyle's project:
   https://github.com/9re/android-support-v4-googlemaps

And yes, I also searched a solution for all this issue, and even providing the second way too.
Searched, tried to implement my ones, and so on. As long as i just found Kobayashi ones ;D So, it just remains to use ? Oh no, nothing is as easy as it sounds.

"Hey Blondie! You know what you are? Just a dirty son of a ahAHahAHaaaaah!"

As already shown in petedoyle's project, there is some issue with MapFragment, especially instancing MapView:
https://github.com/petedoyle/android-support-v4-googlemaps/blob/master/samples/FragmentLayoutMaps/src/com/example/android/apis/app/FragmentLayout.java plz focus on "onCreateView", row 224 (and row 80, for fields declarations)


As you could see, Pete instances a View inflating from R.layout.map, that is file containing <com.google.android.maps.MapView> (or another MapView), and then this view is returned as viewcontainer in onCreateView.
In onCreateView body he use mapview and does his stuffs, and you can see as View mapView and View mapViewContainer are instanced in parent FragmentMapActivity. But why ?

Fragment borns to contains all ui and event code, and handles fields life within its lifecycle, so we would instance our fields in our MapFragment, not in FragmentMapActivity.

Then, my endeavour (anyhow, it works) for MapFragment:

• it handles viewContainer directly, instancing it in onAttach;
• it is an abstract class, with "providesMapViewContainerId" abstract method;
  in concrete subclass, it provides file layout resource id
• a concrete (final?) "addMapViewToViewParent" method, to be invoked within concrete subclass onCreateView, passing View parameter as its first parameter, and our MapView as second ones.

And MapView where it comes from?
Simply with findViewById (or injecting as singleton, if you use roboguice). So:

public MyMapFragment extends MapFragment {
   private MapView mapView;
   /*
    * other methods
    */
   @Override
   public void onViewCreated(View view, Bundle savedInstanceState) {
      super.onViewCreated(view, savedInstanceState);
      if (mapView == null) // cause only mapview is admitted
         mapView = getActivity().findViewById(R.id.yourmapview);
      addMapViewToViewParent(view,mapView);

      // your stuffs with mapview
   }
}

and MapFragment.java on github or code below:

/*
 * Copyright (C) 2012 Massimiliano Leone - maximilianus@gmail.com
 * Licensed under the Apache License, Version 2.0 (the "License");
 */
package android.support.v4.app;

import android.app.Activity;
import android.os.Bundle;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.FrameLayout;

import com.google.android.maps.MapView;

public abstract class MapFragment extends Fragment {

  private View mapViewContainer;

  /**
    * inflating from layout provided by (your concrete) providesMapViewContainerId method;
    * here we instance the viewcontainer returned by onViewCreated,
    * something as:
    *
    * <pre>return LayoutInflater.from(activity).inflate(R.layout.yourmaplayout, null);* </pre>
    */
  @Override
  public void onAttach(Activity activity) {
    super.onAttach(activity);       

    if (mapViewContainer == null) {
      mapViewContainer =  LayoutInflater.from(activity).inflate( providesMapViewContainerId(), null);
    }
  }

  /**
    * in your concrete class you have to provide the layout containing mapview:
    * something as:
    *
    * "return R.layout.map;"
    * where R.layout.map is map.xml file in layout directory, declaring a RelativeLayout,
    * which contains Mapview, as below:
    *
    * <pre>
    *  <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    *   android:id="@+id/maplayout"
    *   android:orientation="vertical"
    *   android:layout_width="fill_parent"
    *   android:layout_height="fill_parent">

    *  <com.google.android.maps.MapView
    *    android:id="@+id/mapview"
    *    android:layout_width="fill_parent"
    *    android:layout_height="fill_parent"
    *    android:clickable="true"
    *    android:apiKey="YOUR_APIKEY"/>
    *  </RelativeLayout>
    * </pre>    
    * @param activity
    * @return View
    */
  protected abstract int providesMapViewContainerId();

  protected void addMapViewToViewParent(View view, MapView mapView) {

    ViewGroup viewGroup = (ViewGroup) view.getParent();
    if (viewGroup == null) viewGroup = (ViewGroup) view;
    int childs = viewGroup.getChildCount();
    if (childs == 0) {
 viewGroup.addView(mapView);
    } else {
 for (int i = 0; i < viewGroup.getChildCount(); i++) {
   View child = viewGroup.getChildAt(i);
   if (child instanceof FrameLayout) {
       continue;
   } else if (child instanceof MapView) {
       viewGroup.removeView(child);
       viewGroup.addView(mapView, 1);
   } else {
       viewGroup.removeView(child);
   }
      }
      //viewGroup.addView(mapView, 0);
    }
    viewGroup.invalidate();
  }   

  @Override
  public View onCreateView(LayoutInflater inflater, ViewGroup container,Bundle savedInstanceState) {
  return mapViewContainer;
  }

  @Override
  public void onDestroyView() {
    super.onDestroyView();
    ViewGroup parentViewGroup = (ViewGroup) mapViewContainer.getParent();
    if( parentViewGroup != null ) {
      parentViewGroup.removeView( mapViewContainer );
      //parentViewGroup.removeAllViews();
    }
  }   

}

some android ui tools and tutorial

http://android-ui-utils.googlecode.com/hg/asset-studio/dist/icons-actionbar.html#source.space.trim=1&source.space.pad=0&name=sort&theme=light

http://blogfreakz.com/mobile-development/android-gui-icons-and-fonts/

http://developer.android.com/design/style/color.html

http://developer.android.com/guide/practices/ui_guidelines/index.html

nice presentation about roboguice

http://www.blog.project13.pl/wp-content/uploads/2011/12/presentation.html#slide1

Android - Creating custom button assets

Android Developers — Visual Design Screencast — Creating custom button assets - YouTube:

for creating android button

android usb reverse tethering via code

welcome again !

the problem of day is:
when i enable usb reverse tethering adb doesn't not respond anymore, via usb.

so, to have all working (reverse tethering, adb, etc) we need many operations:
1) enable reverse tethering via settings
2) change adb prop to enable tcp
3) start a dhcp request on usb0
4) disable wifi
5) other annoying stuffs?

(and reverse, of course :/)

so, it's time to find an -ultimate- solution.

1) with LLAMA (don't u know LLAMA? know it ! https://market.android.com/details?id=com.kebab.Llama ) i can start a "enable reverse tethering and so others stuff" when i plug the cable (and when certain others conditions are verified - for example: i was in wifi, so i want again connectivity).

2) with irssi connectbot (https://play.google.com/store/apps/details?id=org.woltage.irssiconnectbot) i can start some script which call dhcpd and change adb prop (just a "adb_tcp" script to enable adb over tcp, and "adb_usb" to rechange over usb)

download https://raw.github.com/k0smik0/burt/master/scripts/adb_tcp

#!/system/bin/sh

setprop service.adb.tcp.port 5555
stop adbd
start adbd

and download https://raw.github.com/k0smik0/burt/master/scripts/adb_usb

#!/system/bin/sh

setprop service.adb.tcp.port -1
stop adbd
start adbd

and "netcfg usb0 dhcp" to start a dhcp request (plz attention you need ROOTED phone in order to execute this last one - in general, a rooted phone is needed for all the tricks, actually...)

3) last thing (but it's the worst problem) is to enable usb tethering, and i don't want to check via settings, so it's time to code some lines.

the idea beyond is just simple:
send a broadcast message and a receiver intercepts it and enables/disables usb tethering.
Below, the receiver:

/**
 * GPL Copyleft 2012 Massimiliano Leone - maximilianus@gmail.com .
 */
package net.iubris.burt;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;

public class UsbReverseTetheringIncomingReceiver extends BroadcastReceiver {

  protected static final String ACTION_START = "net.iubris.burt.START";
  protected static final String ACTION_STOP = "net.iubris.burt.STOP";
 
  @Override
  public void onReceive(Context context, Intent intent) {
    final String action = intent.getAction();  
    final Object obj = context.getSystemService(Context.CONNECTIVITY_SERVICE);
    if (action.equals(ACTION_START)) {
      actOnTether(obj, "tether");
    } else if (action.equals(ACTION_STOP)) {
      actOnTether(obj, "untether");
    }
  }
    
  private final void actOnTether(Object obj, String act) {
    for (Method m : obj.getClass().getDeclaredMethods()) {
      if (m.getName().equals(act)) {
 try {
   m.invoke(obj, "usb0");
 } catch (IllegalArgumentException e) {
   e.printStackTrace();
 } catch (IllegalAccessException e) {
   e.printStackTrace();
 } catch (InvocationTargetException e) {
   e.printStackTrace();
 }
      }
    }
  } 
}

and the relative AndroidManifest.xml with receiver declaration, permissions, etc.

<manifest android:versioncode="1" android:versionname="1.0" package="net.iubris.burt" xmlns:android="http://schemas.android.com/apk/res/android">

  <uses-sdk android:minsdkversion="8">
    
  <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE">
  <uses-permission android:name="android.permission.CHANGE_NETWORK_STATE">

  <application android:debuggable="false" android:icon="@drawable/icon" android:label="@string/app_name">
    <activity android:label="@string/app_name" android:name=".UsbReverseTetheringActivity">
      <intent-filter>
 <action android:name="android.intent.action.MAIN">
   <category android:name="android.intent.category.LAUNCHER"/>
 </action>
      </intent-filter>
    </activity>
    <receiver android:enabled="true" android:name=".UsbReverseTetheringIncomingReceiver">
      <intent-filter>
 <action android:name="net.iubris.burt.START"/>
 <action android:name="net.iubris.burt.STOP"/>
      </intent-filter>
    </receiver>
  </application>
</manifest>

ok, there is also an activity, with 2 buttons start/stop.. but just for a visual sample:
actually, into code, the buttons send broadcast "net.iubris.burt.START" and "net.iubris.burt.STOP"

so, in terminal (irssi connectbot? another else?) the commands are just:

am broadcast -a "net.iubris.burt.START" # to enable
am broadcast -a "net.iubris.burt.STOP" # to disable

and obviously via adb: "adb shell am broadcast etc."

4) finally, a summa cum laude of all these pieces :
--- open irssi connectbot ---
- create a "local" connection called "start usb reverse tethering" (or as u prefer)
- edit its "post-login automation" with these lines (careful to return or use && !)

am broadcast -a net.iubris.burt.START
su -c "netcfg usb0 dhcp"
su -c "adb_tcp"

adb_tcp (from 1, above) is just a script, so u can replace last line with:

su -c "setprop service.adb.tcp.port 5555 && stop adbd && start adbd"

in some terminals (due to kernel/rom version) this last ones is not needed, because adb over usb is working also with usb reverse tethering


--- open Llama ---
- create new event and call "start usb reverse tethering" (or as u want)
- add condition "charging status" as "charging from usb" and other conditions u want it matches (i.e. i have "wifi network connected" = "on" so this switch from wifi to usb r.t. does its jobs because phone is really in a state which we can call "connected")
- add actions: "toggle wifi" as "wifi off" and "run app shortcut" as [irssiconnectbot action], that is the irssiconnectbot local connection u specified above, "start usb reverse tethering" (or name you used for)


5) and to restore ?
--- for irssi-connectbot, a new local connection called "stop usb reverse tethering", where "post-login automation" is:

am broadcast -a net.iubris.burt.STOP
su -c "adb_usb"
as for "adb_tcp", the "adb_usb" is just these commands:
su -c "setprop service.adb.tcp.port -1 && stop adbd && start adbd"

--- and in LLama, an event for "stop usb reverse tethering" can be something like where match condition "charging status" = "using battery" and action "run app shortcut" is irssi-connectbot "stop usb reverse tethering"



finally, the "burt" receiver project on github burt and the builder apk burt.apk



credits to:
http://tjandroid.blogspot.com/2010/12/enabledisable-usb-tethering.html
http://www.apad.tv/apadforum/showthread.php?1280-Solved-system-bin-sh-am-not-found-and-system-bin-sh-pm-not-found

http://www.irasenthil.com/2011/09/how-to-inject-context-in-android-with.html?utm_medium=twitter&utm_source=twitterfeed

injecting context using roboguice

roboguice, androidannotations, ignition, rest - click here